Sep 11 17:53:56.103587 ip-10-0-133-109 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 17:53:56.550845 ip-10-0-133-109 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:53:56.550845 ip-10-0-133-109 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 17:53:56.550845 ip-10-0-133-109 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:53:56.550845 ip-10-0-133-109 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 17:53:56.550845 ip-10-0-133-109 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:53:56.551920 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.551837    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 17:53:56.555798 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555783    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:53:56.555798 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555798    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555801    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555805    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555808    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555810    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555813    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555815    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555818    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555821    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555824    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555826    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555829    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555832    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555834    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555837    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555840    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555843    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555845    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555848    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555850    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:53:56.555862 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555853    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555856    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555858    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555861    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555863    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555866    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555869    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555872    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555875    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555878    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555880    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555883    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555885    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555888    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555890    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555893    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555895    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555898    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555900    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555903    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:53:56.556379 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555905    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555908    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555910    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555913    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555915    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555920    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555923    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555927    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555931    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555935    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555938    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555940    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555943    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555945    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555949    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555952    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555956    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555959    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:53:56.556885 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555962    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555964    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555967    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555969    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555972    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555974    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555977    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555980    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555982    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555985    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555987    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555990    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555993    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555996    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.555998    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556001    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556003    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556006    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556008    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556010    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:53:56.557340 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556013    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556016    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556019    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556022    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556024    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556027    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556029    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556032    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556035    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556037    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556040    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556043    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556045    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556048    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556050    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556053    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556055    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556058    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556060    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.556063    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:53:56.557837 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557732    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557739    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557743    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557746    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557748    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557752    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557754    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557757    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557760    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557762    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557765    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557768    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557770    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557773    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557776    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557779    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557782    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557784    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557787    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557789    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:53:56.558439 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557792    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557795    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557798    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557801    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557803    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557806    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557808    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557811    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557813    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557816    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557818    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557821    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557824    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557826    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557829    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557831    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557834    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557836    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557839    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557842    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:53:56.558984 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557847    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557851    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557854    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557858    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557860    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557863    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557866    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557869    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557872    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557874    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557877    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557879    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557882    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557884    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557887    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557890    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557892    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557895    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557897    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:53:56.559510 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557900    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557904    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557907    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557910    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557913    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557916    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557919    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557921    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557923    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557926    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557928    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557931    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557934    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557937    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557939    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557942    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557945    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557948    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557952    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:53:56.559964 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557956    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557959    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557962    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557965    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557968    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557970    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557974    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557976    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557979    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557982    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557984    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557987    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557989    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557991    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557994    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557997    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.557999    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558002    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558004    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:53:56.560441 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558007    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558009    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558382    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558388    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558393    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558396    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558400    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558403    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558406    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558409    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558411    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558414    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558417    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558420    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558423    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558426    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558429    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558431    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558434    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:53:56.560908 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558437    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558440    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558443    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558446    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558448    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558451    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558454    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558457    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558459    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558462    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558465    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558467    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558470    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558473    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558475    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558478    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558480    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558483    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558485    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:53:56.561400 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558488    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558493    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558497    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558500    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558503    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558505    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558508    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558511    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558513    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558516    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558519    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558521    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558524    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558526    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558529    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558531    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558534    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558538    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558541    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:53:56.561867 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558543    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558546    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558548    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558551    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558554    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558556    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558559    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558561    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558564    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558566    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558568    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558571    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558574    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558577    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558579    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558582    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558585    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558588    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558590    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558593    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:53:56.562339 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558595    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558598    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558601    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558603    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558606    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558608    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558611    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558614    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558616    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558619    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558621    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558624    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558627    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558629    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558632    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558634    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558637    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558639    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558642    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558644    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:53:56.562830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558646    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558649    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558651    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.558654    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558726    2571 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558769    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558779    2571 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558783    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558788    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558791    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558795    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558801    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558805    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558808    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558811    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558814    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558817    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558820    2571 flags.go:64] FLAG: --cgroup-root=""
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558823    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558826    2571 flags.go:64] FLAG: --client-ca-file=""
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558829    2571 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558832    2571 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558836    2571 flags.go:64] FLAG: --cluster-domain=""
Sep 11 17:53:56.563327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558838    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558841    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558844    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558848    2571 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558852    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558854    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558857    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558860    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558864    2571 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558866    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558870    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558874    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558877    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558881    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558885    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558887    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558890    2571 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558893    2571 flags.go:64] FLAG: --enable-server="true"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558897    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558902    2571 flags.go:64] FLAG: --event-burst="100"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558907    2571 flags.go:64] FLAG: --event-qps="50"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558910    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558913    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558916    2571 flags.go:64] FLAG: --eviction-hard=""
Sep 11 17:53:56.563893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558920    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558923    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558926    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558928    2571 flags.go:64] FLAG: --eviction-soft=""
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558931    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558934    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558937    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558940    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558943    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558946    2571 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558949    2571 flags.go:64] FLAG: --feature-gates=""
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558952    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558955    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558958    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558961    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558964    2571 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558967    2571 flags.go:64] FLAG: --help="false"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558970    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558973    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558977    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558980    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558983    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558987    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558990    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 17:53:56.564524 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558993    2571 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558996    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.558999    2571 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559003    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559009    2571 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559012    2571 flags.go:64] FLAG: --kube-reserved=""
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559014    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559017    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559020    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559023    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559026    2571 flags.go:64] FLAG: --lock-file=""
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559029    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559032    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559035    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559040    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559043    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559045    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559048    2571 flags.go:64] FLAG: --logging-format="text"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559051    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559054    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559057    2571 flags.go:64] FLAG: --manifest-url=""
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559060    2571 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559064    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559067    2571 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559070    2571 flags.go:64] FLAG: --max-pods="110"
Sep 11 17:53:56.565142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559073    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559076    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559080    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559083    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559086    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559089    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559092    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559099    2571 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559102    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559105    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559108    2571 flags.go:64] FLAG: --pod-cidr=""
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559112    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559118    2571 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559121    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559124    2571 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559127    2571 flags.go:64] FLAG: --port="10250"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559130    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559132    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-026e3a457f7d9706a"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559136    2571 flags.go:64] FLAG: --qos-reserved=""
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559138    2571 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559141    2571 flags.go:64] FLAG: --register-node="true"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559144    2571 flags.go:64] FLAG: --register-with-taints=""
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559153    2571 flags.go:64] FLAG: --registry-burst="10"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559156    2571 flags.go:64] FLAG: --registry-qps="5"
Sep 11 17:53:56.565779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559159    2571 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559161    2571 flags.go:64] FLAG: --reserved-memory=""
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559165    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559168    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559171    2571 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559174    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559177    2571 flags.go:64] FLAG: --runonce="false"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559180    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559183    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559186    2571 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559190    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559193    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559196    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559200    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559203    2571 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559206    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559209    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559211    2571 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559214    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559217    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559222    2571 flags.go:64] FLAG: --system-cgroups=""
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559225    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559230    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559233    2571 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559236    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 17:53:56.566387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559240    2571 flags.go:64] FLAG: --tls-min-version=""
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559242    2571 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559245    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559248    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559251    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559254    2571 flags.go:64] FLAG: --v="2"
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559258    2571 flags.go:64] FLAG: --version="false"
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559261    2571 flags.go:64] FLAG: --vmodule=""
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559265    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.559268    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559390    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559394    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559397    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559400    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559403    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559406    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559410    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559414    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559417    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559420    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559422    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559425    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559428    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:53:56.567035 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559430    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559433    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559436    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559439    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559443    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559446    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559449    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559452    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559454    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559457    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559460    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559462    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559465    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559468    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559470    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559474    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559478    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559481    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559484    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:53:56.567607 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559487    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559489    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559492    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559495    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559499    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559502    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559505    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559510    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559513    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559517    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559520    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559523    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559526    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559529    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559531    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559534    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559537    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559540    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559543    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:53:56.568111 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559546    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559549    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559552    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559554    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559557    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559559    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559562    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559565    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559567    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559570    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559572    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559575    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559577    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559580    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559582    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559585    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559587    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559590    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559593    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559596    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:53:56.568597 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559599    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559601    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559604    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559606    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559609    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559611    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559614    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559617    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559620    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559622    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559625    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559628    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559630    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559633    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559636    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559646    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559649    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559651    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559654    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559657    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:53:56.569085 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559660    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559662    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559665    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559667    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559670    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559672    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559675    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.559679    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.560389    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.568016    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.568029    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.568905    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.568913    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.568982    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:53:56.569601 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.568988    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.568993    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.568996    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.568999    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569003    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569006    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569008    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569011    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569015    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569018    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569020    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569023    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569026    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569029    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569032    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569035    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569038    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569040    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569043    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569046    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:53:56.569940 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569048    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569051    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569053    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569056    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569058    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569061    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569064    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569066    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569070    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569073    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569075    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569078    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569081    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569083    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569086    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569089    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569093    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569097    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569100    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:53:56.570446 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569102    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569105    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569108    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569111    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569113    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569116    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569118    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569121    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569124    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569126    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569129    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569131    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569134    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569136    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569139    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569142    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569144    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569147    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569150    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569153    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:53:56.570902 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569155    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569158    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569161    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569163    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569166    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569168    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569173    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569176    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569179    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569181    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569184    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569187    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569190    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569192    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569195    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569198    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569201    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569203    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569206    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569209    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:53:56.571414 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569212    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569214    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569217    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569219    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569222    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569224    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569227    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569229    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569232    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569234    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569237    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569240    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569242    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569245    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569247    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569251    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569253    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569256    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:53:56.571936 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569259    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.569263    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569402    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569408    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569411    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569414    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569417    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569420    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569422    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569424    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569427    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569429    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569432    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569434    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569437    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:53:56.572481 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569439    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569442    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569444    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569447    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569449    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569452    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569455    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569457    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569461    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569464    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569467    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569470    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569473    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569477    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569480    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569483    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569486    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569489    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569492    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:53:56.572843 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569495    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569497    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569500    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569503    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569505    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569508    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569510    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569513    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569515    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569518    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569520    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569524    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569527    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569531    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569534    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569537    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569540    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569543    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569545    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:53:56.573487 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569548    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569550    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569553    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569555    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569558    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569560    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569563    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569565    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569568    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569570    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569574    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569576    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569579    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569581    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569583    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569586    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569589    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569591    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569593    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569596    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:53:56.573988 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569598    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569601    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569604    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569607    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569609    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569612    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569614    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569617    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569620    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569623    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569625    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569628    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569630    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569632    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569635    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569637    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569640    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569642    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569645    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569647    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:53:56.574519 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569650    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569653    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569655    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569658    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569660    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569663    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569665    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:56.569668    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.569673    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.569769    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.572002    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.573012    2571 server.go:1019] "Starting client certificate rotation"
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.573125    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:53:56.575161 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.573186    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:53:56.578122 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.578102    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:53:56.579971 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.579953    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:53:56.589829 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.589812    2571 log.go:25] "Validated CRI v1 runtime API"
Sep 11 17:53:56.593024 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.593010    2571 log.go:25] "Validated CRI v1 image API"
Sep 11 17:53:56.593833 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.593813    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 17:53:56.599980 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.599958    2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b6ea9cf6-f3af-452f-916e-fab305cb9204:/dev/nvme0n1p3 e3f89ba9-58d2-4141-bbdf-77ca2e9bb730:/dev/nvme0n1p4]
Sep 11 17:53:56.600046 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.599981    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 17:53:56.605500 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.605385    2571 manager.go:216] Machine: {Timestamp:2026-09-11 17:53:56.603872798 +0000 UTC m=+0.373939144 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3148313 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20da3dae12ed5f060cdcf856f4d714 SystemUUID:ec20da3d-ae12-ed5f-060c-dcf856f4d714 BootID:a177c229-7379-44a1-b9df-f32feb664c3f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7a:df:1a:29:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7a:df:1a:29:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:b4:df:3f:d8:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 17:53:56.605500 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.605488    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 17:53:56.605616 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.605568    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 17:53:56.609041 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.609015    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 17:53:56.609173 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.609044    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-109.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 17:53:56.609848 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.609837    2571 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 17:53:56.609885 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.609850    2571 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 17:53:56.609885 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.609859    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:53:56.609885 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.609873    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:53:56.610198 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.610183    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:53:56.611189 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.611178    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 17:53:56.615810 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.615791    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:53:56.615914 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.615904    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 17:53:56.615961 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.615935    2571 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 17:53:56.615961 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.615948    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 17:53:56.616015 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.615966    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 17:53:56.616015 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.615981    2571 kubelet.go:398] "Adding apiserver pod source"
Sep 11 17:53:56.616015 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.615992    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 17:53:56.616432 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.616420    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:53:56.616481 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.616445    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:53:56.620421 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.620396    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 17:53:56.623401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.623386    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 17:53:56.623457 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.623417    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 17:53:56.626195 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626171    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 17:53:56.626195 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626192    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 17:53:56.626195 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626199    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626207    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626219    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626228    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626247    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626256    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626265    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626273    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626285    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 17:53:56.626401 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.626310    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 17:53:56.627217 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.627204    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 17:53:56.627217 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.627214    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 17:53:56.633595 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.633570    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 17:53:56.633595 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.633571    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 17:53:56.636218 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.636034    2571 server.go:1268] "Started kubelet"
Sep 11 17:53:56.636290 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.636111    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 17:53:56.636385 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.636122    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 17:53:56.636440 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.636416    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 17:53:56.637197 ip-10-0-133-109 systemd[1]: Started Kubernetes Kubelet.
Sep 11 17:53:56.637492 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.637401    2571 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 17:53:56.638168 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.637962    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 17:53:56.639206 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.639186    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-109.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 17:53:56.642671 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.642650    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 17:53:56.646020 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.646002    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 17:53:56.646735 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.646719    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 17:53:56.646834 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.646805    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:56.647490 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647474    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 17:53:56.647579 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647530    2571 factory.go:55] Registering systemd factory
Sep 11 17:53:56.647579 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647543    2571 factory.go:223] Registration of the systemd container factory successfully
Sep 11 17:53:56.647760 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647746    2571 factory.go:153] Registering CRI-O factory
Sep 11 17:53:56.647760 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647761    2571 factory.go:223] Registration of the crio container factory successfully
Sep 11 17:53:56.647913 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647810    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 17:53:56.647913 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647827    2571 factory.go:103] Registering Raw factory
Sep 11 17:53:56.647913 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647842    2571 manager.go:1196] Started watching for new ooms in manager
Sep 11 17:53:56.648049 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647945    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 17:53:56.648049 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.647958    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 17:53:56.648153 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.648098    2571 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 17:53:56.648153 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.648105    2571 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 17:53:56.648309 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.648281    2571 manager.go:349] Starting recovery of all containers
Sep 11 17:53:56.651764 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.651731    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 17:53:56.651865 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.651801    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 17:53:56.653040 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.651881    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-109.ec2.internal.18d455c05bd9962c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-109.ec2.internal,UID:ip-10-0-133-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-109.ec2.internal,},FirstTimestamp:2026-09-11 17:53:56.63599774 +0000 UTC m=+0.406064079,LastTimestamp:2026-09-11 17:53:56.63599774 +0000 UTC m=+0.406064079,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-109.ec2.internal,}"
Sep 11 17:53:56.658500 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.658480    2571 manager.go:354] Recovery completed
Sep 11 17:53:56.662248 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.662225    2571 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 11 17:53:56.666402 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.666388    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:53:56.668191 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.668166    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:53:56.668277 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.668206    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:53:56.668277 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.668217    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:53:56.668821 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.668807    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 17:53:56.668821 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.668820    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 17:53:56.668905 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.668836    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:53:56.668905 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.668856    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:53:56.669581 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.669565    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:53:56.669640 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.669596    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:53:56.669640 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.669609    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:53:56.671042 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.671024    2571 policy_none.go:49] "None policy: Start"
Sep 11 17:53:56.671042 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.671044    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 17:53:56.671158 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.671055    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 17:53:56.671158 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.670996    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-109.ec2.internal.18d455c05dc4d585  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-109.ec2.internal,UID:ip-10-0-133-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-109.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-109.ec2.internal,},FirstTimestamp:2026-09-11 17:53:56.668192133 +0000 UTC m=+0.438258478,LastTimestamp:2026-09-11 17:53:56.668192133 +0000 UTC m=+0.438258478,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-109.ec2.internal,}"
Sep 11 17:53:56.673120 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.673108    2571 policy_none.go:47] "Start"
Sep 11 17:53:56.680886 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.680828    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-109.ec2.internal.18d455c05dc521e3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-109.ec2.internal,UID:ip-10-0-133-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-109.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-109.ec2.internal,},FirstTimestamp:2026-09-11 17:53:56.668211683 +0000 UTC m=+0.438278021,LastTimestamp:2026-09-11 17:53:56.668211683 +0000 UTC m=+0.438278021,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-109.ec2.internal,}"
Sep 11 17:53:56.684805 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.684791    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qtm46"
Sep 11 17:53:56.691013 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.690924    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-109.ec2.internal.18d455c05dc546c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-109.ec2.internal,UID:ip-10-0-133-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-109.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-109.ec2.internal,},FirstTimestamp:2026-09-11 17:53:56.668221127 +0000 UTC m=+0.438287466,LastTimestamp:2026-09-11 17:53:56.668221127 +0000 UTC m=+0.438287466,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-109.ec2.internal,}"
Sep 11 17:53:56.693427 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.693408    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qtm46"
Sep 11 17:53:56.710070 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.710056    2571 manager.go:337] "Starting Device Plugin manager"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.710095    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.710108    2571 server.go:87] "Starting device plugin registration server"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.710416    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.710426    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.710514    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.710614    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.710625    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.711093    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.711142    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.722169    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.723681    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.723730    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.723759    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.723802    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 17:53:56.728927 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.725502    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:53:56.810964 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.810863    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:53:56.811839 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.811820    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:53:56.811932 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.811851    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:53:56.811932 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.811861    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:53:56.811932 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.811886    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.820874 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.820856    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.820937 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.820881    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-109.ec2.internal\": node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:56.824033 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.824017    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"]
Sep 11 17:53:56.824079 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.824073    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:53:56.824891 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.824874    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:53:56.824983 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.824907    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:53:56.824983 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.824921    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:53:56.827229 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827214    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:53:56.827355 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827338    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.827407 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827365    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:53:56.827903 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827885    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:53:56.827997 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827911    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:53:56.827997 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827926    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:53:56.827997 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827886    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:53:56.827997 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827980    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:53:56.827997 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.827995    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:53:56.830249 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.830233    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.830342 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.830261    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:53:56.830910 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.830897    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:53:56.830978 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.830924    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:53:56.830978 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.830941    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:53:56.843921 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.843895    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-109.ec2.internal\" not found" node="ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.848465 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.848448    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-109.ec2.internal\" not found" node="ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.849485 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.849466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d161ea248816b49b79889a6c4d45f47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal\" (UID: \"8d161ea248816b49b79889a6c4d45f47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.849562 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.849493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d161ea248816b49b79889a6c4d45f47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal\" (UID: \"8d161ea248816b49b79889a6c4d45f47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.849562 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.849511    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90fea5d757634326f5a2e77bf4cd7804-config\") pod \"kube-apiserver-proxy-ip-10-0-133-109.ec2.internal\" (UID: \"90fea5d757634326f5a2e77bf4cd7804\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.850312 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.850287    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:56.950608 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:56.950582    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:56.950608 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.950601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d161ea248816b49b79889a6c4d45f47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal\" (UID: \"8d161ea248816b49b79889a6c4d45f47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.950790 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.950625    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d161ea248816b49b79889a6c4d45f47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal\" (UID: \"8d161ea248816b49b79889a6c4d45f47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.950790 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.950642    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90fea5d757634326f5a2e77bf4cd7804-config\") pod \"kube-apiserver-proxy-ip-10-0-133-109.ec2.internal\" (UID: \"90fea5d757634326f5a2e77bf4cd7804\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.950790 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.950689    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d161ea248816b49b79889a6c4d45f47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal\" (UID: \"8d161ea248816b49b79889a6c4d45f47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.950790 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.950694    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/90fea5d757634326f5a2e77bf4cd7804-config\") pod \"kube-apiserver-proxy-ip-10-0-133-109.ec2.internal\" (UID: \"90fea5d757634326f5a2e77bf4cd7804\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:56.950790 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:56.950730    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8d161ea248816b49b79889a6c4d45f47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal\" (UID: \"8d161ea248816b49b79889a6c4d45f47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:57.050962 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.050924    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.147513 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.147442    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:57.151161 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.151142    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.152713 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.152697    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:57.251941 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.251903    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.353023 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.352989    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.454165 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.454134    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.554230 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.554196    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.573039 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.573009    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 17:53:57.573194 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.573143    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:53:57.654775 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.654740    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.698957 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.698903    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:48:57 +0000 UTC" deadline="2028-04-18 02:14:59.941308352 +0000 UTC"
Sep 11 17:53:57.698957 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.698948    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14024h21m2.24236552s"
Sep 11 17:53:57.746598 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.746573    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:53:57.748442 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.748424    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:53:57.748533 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.748457    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:53:57.748533 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.748469    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:53:57.755758 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.755737    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.850090 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:57.850061    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d161ea248816b49b79889a6c4d45f47.slice/crio-277cc3121dcf0db5d9742c243769dcc76bc590fc3df711470b152fca9da2010f WatchSource:0}: Error finding container 277cc3121dcf0db5d9742c243769dcc76bc590fc3df711470b152fca9da2010f: Status 404 returned error can't find the container with id 277cc3121dcf0db5d9742c243769dcc76bc590fc3df711470b152fca9da2010f
Sep 11 17:53:57.854900 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:57.854873    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fea5d757634326f5a2e77bf4cd7804.slice/crio-52765ef0570fd47ecea72c30b6576b3071dd79e189ff4f478c38d46be5fb3e3f WatchSource:0}: Error finding container 52765ef0570fd47ecea72c30b6576b3071dd79e189ff4f478c38d46be5fb3e3f: Status 404 returned error can't find the container with id 52765ef0570fd47ecea72c30b6576b3071dd79e189ff4f478c38d46be5fb3e3f
Sep 11 17:53:57.855794 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.855783    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.856278 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.856263    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:53:57.891899 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.891872    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:53:57.955990 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:57.955948    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:57.985705 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:57.985643    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:53:58.056679 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.056640    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-109.ec2.internal\" not found"
Sep 11 17:53:58.077966 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.077944    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:53:58.147632 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.147604    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:58.159987 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.159966    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:53:58.161157 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.161145    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal"
Sep 11 17:53:58.176671 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.176652    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:53:58.616804 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.616769    2571 apiserver.go:52] "Watching apiserver"
Sep 11 17:53:58.630506 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.629482    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 17:53:58.630506 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.630049    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal","openshift-dns/node-resolver-l8kwh","openshift-image-registry/node-ca-j6bg7","openshift-multus/multus-nrr6n","openshift-ovn-kubernetes/ovnkube-node-4v47h","kube-system/konnectivity-agent-cnvwk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79","openshift-cluster-node-tuning-operator/tuned-zd9rf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal","openshift-multus/multus-additional-cni-plugins-jmsqt","openshift-multus/network-metrics-daemon-hl9ct","openshift-network-diagnostics/network-check-target-gkxjh","openshift-network-operator/iptables-alerter-mp22s","kube-system/global-pull-secret-syncer-snnhr"]
Sep 11 17:53:58.632828 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.632805    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.635266 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.635244    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:53:58.635396 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.635370    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:53:58.636283 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.636261    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qf8gv\""
Sep 11 17:53:58.636396 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.636375    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 17:53:58.636531 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.636498    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 17:53:58.636692 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.636379    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 17:53:58.637893 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.637873    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.640494 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.640474    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 17:53:58.640844 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.640813    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-46tn5\""
Sep 11 17:53:58.641041 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.641023    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 17:53:58.641300 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.641283    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 17:53:58.641520 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.641504    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.644031 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.643265    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.644292 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.644266    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 17:53:58.644517 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.644499    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 17:53:58.644765 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.644748    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 17:53:58.644867 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.644853    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 17:53:58.645008 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.644993    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-42pz4\""
Sep 11 17:53:58.645916 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.645895    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.646366    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.646404    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8ctk\""
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.646615    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.646700    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.646767    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.646921    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.647062    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.647648    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.648521    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:53:58.649069 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.648593    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:53:58.650625 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.650390    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 17:53:58.651055 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.650879    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:53:58.651790 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.651354    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.652664 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.652645    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48tgb\""
Sep 11 17:53:58.653424 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.653405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.654876 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.654504    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9xv89\""
Sep 11 17:53:58.654876 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.654722    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 17:53:58.655040 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.655024    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 17:53:58.655877 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.655857    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 17:53:58.656223 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.656203    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 17:53:58.656482 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.656461    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ww77d\""
Sep 11 17:53:58.657944 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.657924    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.658956 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.658932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-var-lib-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659060 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.658970    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovnkube-config\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659060 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.658999    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-modprobe-d\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.659060 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659023    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hf6\" (UniqueName: \"kubernetes.io/projected/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-kube-api-access-p6hf6\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.659060 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659050    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659075    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b5f0490-e3a5-415d-a5b5-8702db1664b3-serviceca\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovn-node-metrics-cert\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659123    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-lib-modules\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659147    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-system-cni-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659172    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-daemon-config\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659196    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-socket-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysctl-conf\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-sys-fs\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659274    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7c9\" (UniqueName: \"kubernetes.io/projected/7b5f0490-e3a5-415d-a5b5-8702db1664b3-kube-api-access-rk7c9\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.659305 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659299    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-cni-netd\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659356    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659386    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysconfig\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659415    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7rf8\" (UniqueName: \"kubernetes.io/projected/9323dca8-d7e0-4a88-b107-cfda2a451f8b-kube-api-access-j7rf8\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659439    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovnkube-script-lib\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-tuned\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-k8s-cni-cncf-io\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659548    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-etc-kubernetes\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659592    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtqx\" (UniqueName: \"kubernetes.io/projected/41a5ee9f-5818-42e5-b540-f16bbc7935a4-kube-api-access-pgtqx\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659650    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-systemd\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659673    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-registration-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-ovn\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysctl-d\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659753    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-cni-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.659831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659786    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-socket-dir-parent\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659810    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-netns\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659838    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5k8\" (UniqueName: \"kubernetes.io/projected/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-kube-api-access-dm5k8\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659882    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-node-log\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-run\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.659960    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-hostroot\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rx28\" (UniqueName: \"kubernetes.io/projected/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-kube-api-access-2rx28\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-slash\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660173    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-env-overrides\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660213    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-tmp\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-kubelet\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660282    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660329    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-run-netns\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-etc-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660455    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660502    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-systemd\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660506    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:53:58.660591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660531    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660557    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-kubelet\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-var-lib-kubelet\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660605    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-conf-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-host\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-multus-certs\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-systemd-units\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660672    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-cnibin\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660687    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41a5ee9f-5818-42e5-b540-f16bbc7935a4-cni-binary-copy\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5f0490-e3a5-415d-a5b5-8702db1664b3-host\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660714    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-log-socket\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660734    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-cni-bin\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660747    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-kubernetes\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660762    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-sys\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660775    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-device-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-cni-bin\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.661226 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.660807    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-cni-multus\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.662123 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.661307    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.662123 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.661332    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:53:58.662123 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.661365    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:53:58.662123 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.661404    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 17:53:58.662123 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.661683    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 17:53:58.662123 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.661865    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:53:58.664043 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.664018    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 17:53:58.665007 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.664339    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rh8gd\""
Sep 11 17:53:58.666739 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.666694    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 17:53:58.666854 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.666834    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sfpw5\""
Sep 11 17:53:58.692928 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.692905    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bcplw"
Sep 11 17:53:58.703974 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.703952    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bcplw"
Sep 11 17:53:58.731592 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.731543    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal" event={"ID":"8d161ea248816b49b79889a6c4d45f47","Type":"ContainerStarted","Data":"277cc3121dcf0db5d9742c243769dcc76bc590fc3df711470b152fca9da2010f"}
Sep 11 17:53:58.733560 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.733535    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal" event={"ID":"90fea5d757634326f5a2e77bf4cd7804","Type":"ContainerStarted","Data":"52765ef0570fd47ecea72c30b6576b3071dd79e189ff4f478c38d46be5fb3e3f"}
Sep 11 17:53:58.749140 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.749115    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 17:53:58.761896 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.761873    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7c9\" (UniqueName: \"kubernetes.io/projected/7b5f0490-e3a5-415d-a5b5-8702db1664b3-kube-api-access-rk7c9\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.762010 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.761909    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-cni-netd\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762010 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.761938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762010 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.761995    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762188 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-cni-netd\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762188 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762052    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysconfig\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.762188 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762085    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-host-slash\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.762188 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysconfig\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.762188 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762137    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.762470 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.762470 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762220    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7rf8\" (UniqueName: \"kubernetes.io/projected/9323dca8-d7e0-4a88-b107-cfda2a451f8b-kube-api-access-j7rf8\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762470 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762245    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762470 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762309    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovnkube-script-lib\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762470 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762355    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-tuned\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.762470 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-k8s-cni-cncf-io\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.762470 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-etc-kubernetes\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.762470 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762458    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtqx\" (UniqueName: \"kubernetes.io/projected/41a5ee9f-5818-42e5-b540-f16bbc7935a4-kube-api-access-pgtqx\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762484    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-systemd\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-registration-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762583    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-k8s-cni-cncf-io\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46bb7121-35c8-46ba-a6ff-885976cea1d1-agent-certs\") pod \"konnectivity-agent-cnvwk\" (UID: \"46bb7121-35c8-46ba-a6ff-885976cea1d1\") " pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762622    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-ovn\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762641    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysctl-d\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762675    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-cni-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-socket-dir-parent\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-netns\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762753    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgpx\" (UniqueName: \"kubernetes.io/projected/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-kube-api-access-9vgpx\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762769    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2279fb0-d7d6-42b2-b779-c9ef928edc88-dbus\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762787    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5k8\" (UniqueName: \"kubernetes.io/projected/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-kube-api-access-dm5k8\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:53:58.762808 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762811    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-node-log\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762834    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-run\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-hostroot\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762864    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rx28\" (UniqueName: \"kubernetes.io/projected/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-kube-api-access-2rx28\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762880    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-slash\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762904    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-env-overrides\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762925    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-tmp\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762963    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-kubelet\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.762990    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763024    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2279fb0-d7d6-42b2-b779-c9ef928edc88-kubelet-config\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763054    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-run-netns\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763106    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-etc-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763132    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-systemd\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763202    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763231    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-kubelet\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.763367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763273    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-var-lib-kubelet\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovnkube-script-lib\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-conf-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763326    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cnibin\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763356    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-host\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-multus-certs\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7bs9\" (UniqueName: \"kubernetes.io/projected/c3685618-7216-470e-af33-3c64fb1618c4-kube-api-access-p7bs9\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-systemd-units\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-cnibin\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41a5ee9f-5818-42e5-b540-f16bbc7935a4-cni-binary-copy\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-iptables-alerter-script\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763532    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5f0490-e3a5-415d-a5b5-8702db1664b3-host\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763553    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-log-socket\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-cni-bin\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-kubernetes\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-sys\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763657    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-device-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.764118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763684    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-cni-bin\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763710    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-cni-multus\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763764    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78m7f\" (UniqueName: \"kubernetes.io/projected/9a9a9e68-6934-4724-8309-b9ef4a98ceba-kube-api-access-78m7f\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763801    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3685618-7216-470e-af33-3c64fb1618c4-hosts-file\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-var-lib-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763858    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovnkube-config\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-modprobe-d\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763898    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hf6\" (UniqueName: \"kubernetes.io/projected/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-kube-api-access-p6hf6\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763922    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b5f0490-e3a5-415d-a5b5-8702db1664b3-serviceca\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763958    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovn-node-metrics-cert\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.763980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-lib-modules\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-system-cni-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764017    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-daemon-config\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-socket-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764087    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 17:53:58.764970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764102    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764113    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-system-cni-dir\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764147    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysctl-conf\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46bb7121-35c8-46ba-a6ff-885976cea1d1-konnectivity-ca\") pod \"konnectivity-agent-cnvwk\" (UID: \"46bb7121-35c8-46ba-a6ff-885976cea1d1\") " pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764239    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3685618-7216-470e-af33-3c64fb1618c4-tmp-dir\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764266    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-sys-fs\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764364    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-sys-fs\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764429    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-kubelet\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764481    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-var-lib-kubelet\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764499    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-run\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-conf-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764582    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-registration-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-multus-certs\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764600    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-slash\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764716    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysctl-d\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764782    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-cni-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764807    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-systemd-units\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764837    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-socket-dir-parent\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.765715 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764870    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-cnibin\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.764885    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-run-netns\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-env-overrides\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765098    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-ovn\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765153    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-node-log\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765160    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-kubelet\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765256    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765323    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-run-netns\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765370    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-etc-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765427    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-run-ovn-kubernetes\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765460    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7b5f0490-e3a5-415d-a5b5-8702db1664b3-host\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765485    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-run-systemd\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.765545    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765549    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-systemd\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-hostroot\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.765629    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs podName:139a5b42-f2cc-4f37-9cac-8e7ab4ab4373 nodeName:}" failed. No retries permitted until 2026-09-11 17:53:59.265594553 +0000 UTC m=+3.035660899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs") pod "network-metrics-daemon-hl9ct" (UID: "139a5b42-f2cc-4f37-9cac-8e7ab4ab4373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765680    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-etc-kubernetes\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765686    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-log-socket\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.766621 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765718    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-socket-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765751    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-host\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-sysctl-conf\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765849    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-host-cni-bin\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-sys\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.765951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-kubernetes\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766004    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9323dca8-d7e0-4a88-b107-cfda2a451f8b-var-lib-openvswitch\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766067    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-cni-bin\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766095    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-modprobe-d\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766103    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-host-var-lib-cni-multus\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7b5f0490-e3a5-415d-a5b5-8702db1664b3-serviceca\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766147    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/41a5ee9f-5818-42e5-b540-f16bbc7935a4-system-cni-dir\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766180    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-lib-modules\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766230    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-device-dir\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766611    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/41a5ee9f-5818-42e5-b540-f16bbc7935a4-multus-daemon-config\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovnkube-config\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.767557 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.766741    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/41a5ee9f-5818-42e5-b540-f16bbc7935a4-cni-binary-copy\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.769312 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.769279    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-etc-tuned\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.769430 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.769398    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-tmp\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.771094 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.771034    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9323dca8-d7e0-4a88-b107-cfda2a451f8b-ovn-node-metrics-cert\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.772570 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.772521    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7c9\" (UniqueName: \"kubernetes.io/projected/7b5f0490-e3a5-415d-a5b5-8702db1664b3-kube-api-access-rk7c9\") pod \"node-ca-j6bg7\" (UID: \"7b5f0490-e3a5-415d-a5b5-8702db1664b3\") " pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.776924 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.776875    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:53:58.777021 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.776933    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:53:58.777021 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.776947    2571 projected.go:196] Error preparing data for projected volume kube-api-access-tvbqf for pod openshift-network-diagnostics/network-check-target-gkxjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:53:58.777126 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.777013    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf podName:c5ffe50c-9f38-4ba1-8c97-8a12861d6273 nodeName:}" failed. No retries permitted until 2026-09-11 17:53:59.276995996 +0000 UTC m=+3.047062338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tvbqf" (UniqueName: "kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf") pod "network-check-target-gkxjh" (UID: "c5ffe50c-9f38-4ba1-8c97-8a12861d6273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:53:58.782252 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.782181    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtqx\" (UniqueName: \"kubernetes.io/projected/41a5ee9f-5818-42e5-b540-f16bbc7935a4-kube-api-access-pgtqx\") pod \"multus-nrr6n\" (UID: \"41a5ee9f-5818-42e5-b540-f16bbc7935a4\") " pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.782747 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.782725    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hf6\" (UniqueName: \"kubernetes.io/projected/940e27d0-1102-4bb5-8b6d-22ec7cafea0c-kube-api-access-p6hf6\") pod \"tuned-zd9rf\" (UID: \"940e27d0-1102-4bb5-8b6d-22ec7cafea0c\") " pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:58.782970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.782942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5k8\" (UniqueName: \"kubernetes.io/projected/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-kube-api-access-dm5k8\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:53:58.785101 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.785074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7rf8\" (UniqueName: \"kubernetes.io/projected/9323dca8-d7e0-4a88-b107-cfda2a451f8b-kube-api-access-j7rf8\") pod \"ovnkube-node-4v47h\" (UID: \"9323dca8-d7e0-4a88-b107-cfda2a451f8b\") " pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.785553 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.785532    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rx28\" (UniqueName: \"kubernetes.io/projected/2e97aa25-b0d2-496c-b858-b38a9f3a9a51-kube-api-access-2rx28\") pod \"aws-ebs-csi-driver-node-5tt79\" (UID: \"2e97aa25-b0d2-496c-b858-b38a9f3a9a51\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.865311 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgpx\" (UniqueName: \"kubernetes.io/projected/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-kube-api-access-9vgpx\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.865311 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865306    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2279fb0-d7d6-42b2-b779-c9ef928edc88-dbus\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.865569 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865359    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2279fb0-d7d6-42b2-b779-c9ef928edc88-kubelet-config\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.865569 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.865569 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865430    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cnibin\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.865569 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865456    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7bs9\" (UniqueName: \"kubernetes.io/projected/c3685618-7216-470e-af33-3c64fb1618c4-kube-api-access-p7bs9\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.865569 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865457    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2279fb0-d7d6-42b2-b779-c9ef928edc88-kubelet-config\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.865569 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865466    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2279fb0-d7d6-42b2-b779-c9ef928edc88-dbus\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:58.865569 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865510    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-iptables-alerter-script\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.865569 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.865559    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:53:58.865970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cnibin\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.865970 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:58.865628    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret podName:e2279fb0-d7d6-42b2-b779-c9ef928edc88 nodeName:}" failed. No retries permitted until 2026-09-11 17:53:59.365607994 +0000 UTC m=+3.135674337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret") pod "global-pull-secret-syncer-snnhr" (UID: "e2279fb0-d7d6-42b2-b779-c9ef928edc88") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:53:58.865970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.865970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865706    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78m7f\" (UniqueName: \"kubernetes.io/projected/9a9a9e68-6934-4724-8309-b9ef4a98ceba-kube-api-access-78m7f\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.865970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865733    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3685618-7216-470e-af33-3c64fb1618c4-hosts-file\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.865970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865778    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-system-cni-dir\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.865970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46bb7121-35c8-46ba-a6ff-885976cea1d1-konnectivity-ca\") pod \"konnectivity-agent-cnvwk\" (UID: \"46bb7121-35c8-46ba-a6ff-885976cea1d1\") " pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:53:58.865970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.865829    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3685618-7216-470e-af33-3c64fb1618c4-tmp-dir\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.866460 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.866436    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-host-slash\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.866549 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.866491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.866549 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.866522    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.866651 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.866634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.866702 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.866664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46bb7121-35c8-46ba-a6ff-885976cea1d1-agent-certs\") pod \"konnectivity-agent-cnvwk\" (UID: \"46bb7121-35c8-46ba-a6ff-885976cea1d1\") " pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:53:58.866756 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.866693    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-iptables-alerter-script\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.867001 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.866934    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-system-cni-dir\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.867078 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.867051    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c3685618-7216-470e-af33-3c64fb1618c4-hosts-file\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.867129 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.867075    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3685618-7216-470e-af33-3c64fb1618c4-tmp-dir\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.867184 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.867152    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.867311 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.867288    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a9a9e68-6934-4724-8309-b9ef4a98ceba-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.867398 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.867362    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-host-slash\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.867707 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.867677    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46bb7121-35c8-46ba-a6ff-885976cea1d1-konnectivity-ca\") pod \"konnectivity-agent-cnvwk\" (UID: \"46bb7121-35c8-46ba-a6ff-885976cea1d1\") " pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:53:58.867793 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.867759    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.868004 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.867966    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a9a9e68-6934-4724-8309-b9ef4a98ceba-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.871256 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.871197    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46bb7121-35c8-46ba-a6ff-885976cea1d1-agent-certs\") pod \"konnectivity-agent-cnvwk\" (UID: \"46bb7121-35c8-46ba-a6ff-885976cea1d1\") " pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:53:58.881224 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.881193    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7bs9\" (UniqueName: \"kubernetes.io/projected/c3685618-7216-470e-af33-3c64fb1618c4-kube-api-access-p7bs9\") pod \"node-resolver-l8kwh\" (UID: \"c3685618-7216-470e-af33-3c64fb1618c4\") " pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:58.882103 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.882074    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78m7f\" (UniqueName: \"kubernetes.io/projected/9a9a9e68-6934-4724-8309-b9ef4a98ceba-kube-api-access-78m7f\") pod \"multus-additional-cni-plugins-jmsqt\" (UID: \"9a9a9e68-6934-4724-8309-b9ef4a98ceba\") " pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:58.882603 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.882569    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgpx\" (UniqueName: \"kubernetes.io/projected/e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0-kube-api-access-9vgpx\") pod \"iptables-alerter-mp22s\" (UID: \"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0\") " pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:58.952712 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.952679    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79"
Sep 11 17:53:58.962930 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.962903    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j6bg7"
Sep 11 17:53:58.965541 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:58.965492    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e97aa25_b0d2_496c_b858_b38a9f3a9a51.slice/crio-0a7fcc6b56f7bb528493e731fdc5471c68a988cfb42c34c0a6b1e94156db76c9 WatchSource:0}: Error finding container 0a7fcc6b56f7bb528493e731fdc5471c68a988cfb42c34c0a6b1e94156db76c9: Status 404 returned error can't find the container with id 0a7fcc6b56f7bb528493e731fdc5471c68a988cfb42c34c0a6b1e94156db76c9
Sep 11 17:53:58.974143 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:58.974118    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5f0490_e3a5_415d_a5b5_8702db1664b3.slice/crio-c78a365caf980d309d83b539af23c70c5521b6bcd9942f5a5a9fcb82b8521694 WatchSource:0}: Error finding container c78a365caf980d309d83b539af23c70c5521b6bcd9942f5a5a9fcb82b8521694: Status 404 returned error can't find the container with id c78a365caf980d309d83b539af23c70c5521b6bcd9942f5a5a9fcb82b8521694
Sep 11 17:53:58.984481 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.984458    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nrr6n"
Sep 11 17:53:58.990705 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.990681    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:53:58.996572 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:58.996545    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a5ee9f_5818_42e5_b540_f16bbc7935a4.slice/crio-6e6fa6633a78f7ec23cd10652da60e8919a067f07965d18f54bc7ce35ee8426a WatchSource:0}: Error finding container 6e6fa6633a78f7ec23cd10652da60e8919a067f07965d18f54bc7ce35ee8426a: Status 404 returned error can't find the container with id 6e6fa6633a78f7ec23cd10652da60e8919a067f07965d18f54bc7ce35ee8426a
Sep 11 17:53:58.996718 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:58.996673    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zd9rf"
Sep 11 17:53:59.004436 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:59.004366    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9323dca8_d7e0_4a88_b107_cfda2a451f8b.slice/crio-ba7cad9be7e2b05f7f01c0468760a5152d962d2f0bae48d47ce8416b787c7e44 WatchSource:0}: Error finding container ba7cad9be7e2b05f7f01c0468760a5152d962d2f0bae48d47ce8416b787c7e44: Status 404 returned error can't find the container with id ba7cad9be7e2b05f7f01c0468760a5152d962d2f0bae48d47ce8416b787c7e44
Sep 11 17:53:59.007541 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.007510    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jmsqt"
Sep 11 17:53:59.008878 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:59.008760    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940e27d0_1102_4bb5_8b6d_22ec7cafea0c.slice/crio-bc4a0e2082578004ff227d716a683f008fee2631064b9f84328238396a04e7db WatchSource:0}: Error finding container bc4a0e2082578004ff227d716a683f008fee2631064b9f84328238396a04e7db: Status 404 returned error can't find the container with id bc4a0e2082578004ff227d716a683f008fee2631064b9f84328238396a04e7db
Sep 11 17:53:59.015895 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.015876    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l8kwh"
Sep 11 17:53:59.019137 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:59.019096    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9a9e68_6934_4724_8309_b9ef4a98ceba.slice/crio-84d11e1cb96e7cbf6d4a82946073cd0b73542bea13ef1df3a937213277181f98 WatchSource:0}: Error finding container 84d11e1cb96e7cbf6d4a82946073cd0b73542bea13ef1df3a937213277181f98: Status 404 returned error can't find the container with id 84d11e1cb96e7cbf6d4a82946073cd0b73542bea13ef1df3a937213277181f98
Sep 11 17:53:59.027487 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.027467    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mp22s"
Sep 11 17:53:59.033678 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.033655    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:53:59.160928 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.160892    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:53:59.269664 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.269628    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:53:59.269844 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:59.269794    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:53:59.269902 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:59.269884    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs podName:139a5b42-f2cc-4f37-9cac-8e7ab4ab4373 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:00.269858825 +0000 UTC m=+4.039925153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs") pod "network-metrics-daemon-hl9ct" (UID: "139a5b42-f2cc-4f37-9cac-8e7ab4ab4373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:53:59.370772 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.370730    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:53:59.370945 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.370797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:53:59.370945 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:59.370908    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:53:59.370945 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:59.370940    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:53:59.371099 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:59.370957    2571 projected.go:196] Error preparing data for projected volume kube-api-access-tvbqf for pod openshift-network-diagnostics/network-check-target-gkxjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:53:59.371099 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:59.370940    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:53:59.371099 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:59.371025    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf podName:c5ffe50c-9f38-4ba1-8c97-8a12861d6273 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:00.371003389 +0000 UTC m=+4.141069733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvbqf" (UniqueName: "kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf") pod "network-check-target-gkxjh" (UID: "c5ffe50c-9f38-4ba1-8c97-8a12861d6273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:53:59.371099 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:53:59.371072    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret podName:e2279fb0-d7d6-42b2-b779-c9ef928edc88 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:00.371056136 +0000 UTC m=+4.141122465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret") pod "global-pull-secret-syncer-snnhr" (UID: "e2279fb0-d7d6-42b2-b779-c9ef928edc88") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:53:59.385803 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:53:59.385773    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3685618_7216_470e_af33_3c64fb1618c4.slice/crio-39244a220385b029b20c9e5cc2a41a6605a4cd158528f0e019ebc3798546d355 WatchSource:0}: Error finding container 39244a220385b029b20c9e5cc2a41a6605a4cd158528f0e019ebc3798546d355: Status 404 returned error can't find the container with id 39244a220385b029b20c9e5cc2a41a6605a4cd158528f0e019ebc3798546d355
Sep 11 17:53:59.708047 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.707998    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:48:59 +0000 UTC" deadline="2028-02-09 08:54:46.355916646 +0000 UTC"
Sep 11 17:53:59.708047 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.708042    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12375h0m46.647880581s"
Sep 11 17:53:59.738125 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.738081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l8kwh" event={"ID":"c3685618-7216-470e-af33-3c64fb1618c4","Type":"ContainerStarted","Data":"39244a220385b029b20c9e5cc2a41a6605a4cd158528f0e019ebc3798546d355"}
Sep 11 17:53:59.739958 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.739663    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" event={"ID":"9a9a9e68-6934-4724-8309-b9ef4a98ceba","Type":"ContainerStarted","Data":"84d11e1cb96e7cbf6d4a82946073cd0b73542bea13ef1df3a937213277181f98"}
Sep 11 17:53:59.741227 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.741193    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zd9rf" event={"ID":"940e27d0-1102-4bb5-8b6d-22ec7cafea0c","Type":"ContainerStarted","Data":"bc4a0e2082578004ff227d716a683f008fee2631064b9f84328238396a04e7db"}
Sep 11 17:53:59.742680 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.742653    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrr6n" event={"ID":"41a5ee9f-5818-42e5-b540-f16bbc7935a4","Type":"ContainerStarted","Data":"6e6fa6633a78f7ec23cd10652da60e8919a067f07965d18f54bc7ce35ee8426a"}
Sep 11 17:53:59.744334 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.744241    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j6bg7" event={"ID":"7b5f0490-e3a5-415d-a5b5-8702db1664b3","Type":"ContainerStarted","Data":"c78a365caf980d309d83b539af23c70c5521b6bcd9942f5a5a9fcb82b8521694"}
Sep 11 17:53:59.746412 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.746341    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79" event={"ID":"2e97aa25-b0d2-496c-b858-b38a9f3a9a51","Type":"ContainerStarted","Data":"0a7fcc6b56f7bb528493e731fdc5471c68a988cfb42c34c0a6b1e94156db76c9"}
Sep 11 17:53:59.748061 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:53:59.748030    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"ba7cad9be7e2b05f7f01c0468760a5152d962d2f0bae48d47ce8416b787c7e44"}
Sep 11 17:54:00.278122 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.278007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:00.278226 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.278206    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:00.278284 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.278274    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs podName:139a5b42-f2cc-4f37-9cac-8e7ab4ab4373 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:02.278254638 +0000 UTC m=+6.048320968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs") pod "network-metrics-daemon-hl9ct" (UID: "139a5b42-f2cc-4f37-9cac-8e7ab4ab4373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:00.379508 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.378527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:00.379508 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.378590    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:00.379508 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.378744    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:00.379508 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.378802    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret podName:e2279fb0-d7d6-42b2-b779-c9ef928edc88 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:02.378781699 +0000 UTC m=+6.148848030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret") pod "global-pull-secret-syncer-snnhr" (UID: "e2279fb0-d7d6-42b2-b779-c9ef928edc88") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:00.379508 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.379307    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:00.379508 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.379375    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:00.379508 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.379429    2571 projected.go:196] Error preparing data for projected volume kube-api-access-tvbqf for pod openshift-network-diagnostics/network-check-target-gkxjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:00.380386 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.380361    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf podName:c5ffe50c-9f38-4ba1-8c97-8a12861d6273 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:02.379460084 +0000 UTC m=+6.149526412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvbqf" (UniqueName: "kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf") pod "network-check-target-gkxjh" (UID: "c5ffe50c-9f38-4ba1-8c97-8a12861d6273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:00.708871 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.708166    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:48:59 +0000 UTC" deadline="2028-02-22 21:28:07.560765737 +0000 UTC"
Sep 11 17:54:00.708871 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.708217    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12699h34m6.852555032s"
Sep 11 17:54:00.727566 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.726636    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:00.727566 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.726762    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:00.727566 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.727168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:00.727566 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.727293    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:00.727566 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.727419    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:00.727566 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:00.727522    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:00.762312 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.762070    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal" event={"ID":"90fea5d757634326f5a2e77bf4cd7804","Type":"ContainerStarted","Data":"b791c7e1e9ec1316a9d1860705cdcec875e649d5dadbf3e6c13d73487ee2ede2"}
Sep 11 17:54:00.765122 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.765039    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cnvwk" event={"ID":"46bb7121-35c8-46ba-a6ff-885976cea1d1","Type":"ContainerStarted","Data":"026820a3aaa45b09188cad22770af9c2be8865b9e9df330e36e069ae1263c5d0"}
Sep 11 17:54:00.773083 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.773031    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mp22s" event={"ID":"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0","Type":"ContainerStarted","Data":"9ba5874ac0640548d1425dbc1ebe0954a15538190510db14ce1a0314c0e281d1"}
Sep 11 17:54:00.793967 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:00.790227    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-109.ec2.internal" podStartSLOduration=2.790206451 podStartE2EDuration="2.790206451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:00.789715929 +0000 UTC m=+4.559782280" watchObservedRunningTime="2026-09-11 17:54:00.790206451 +0000 UTC m=+4.560272807"
Sep 11 17:54:01.790274 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:01.790227    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal" event={"ID":"8d161ea248816b49b79889a6c4d45f47","Type":"ContainerStarted","Data":"4321f64d41ca2c24e1e1f2027251d1c3914b070e9ad7c8c0da1dddfc6c51038a"}
Sep 11 17:54:02.302876 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:02.302279    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:02.302876 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.302466    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:02.302876 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.302530    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs podName:139a5b42-f2cc-4f37-9cac-8e7ab4ab4373 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:06.302514981 +0000 UTC m=+10.072581310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs") pod "network-metrics-daemon-hl9ct" (UID: "139a5b42-f2cc-4f37-9cac-8e7ab4ab4373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:02.403647 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:02.402735    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:02.403647 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:02.402833    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:02.403647 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.403006    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:02.403647 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.403027    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:02.403647 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.403040    2571 projected.go:196] Error preparing data for projected volume kube-api-access-tvbqf for pod openshift-network-diagnostics/network-check-target-gkxjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:02.403647 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.403103    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf podName:c5ffe50c-9f38-4ba1-8c97-8a12861d6273 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:06.4030824 +0000 UTC m=+10.173148745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvbqf" (UniqueName: "kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf") pod "network-check-target-gkxjh" (UID: "c5ffe50c-9f38-4ba1-8c97-8a12861d6273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:02.403647 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.403553    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:02.403647 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.403609    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret podName:e2279fb0-d7d6-42b2-b779-c9ef928edc88 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:06.403592073 +0000 UTC m=+10.173658432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret") pod "global-pull-secret-syncer-snnhr" (UID: "e2279fb0-d7d6-42b2-b779-c9ef928edc88") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:02.729236 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:02.729036    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:02.729236 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:02.729163    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:02.729236 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.729170    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:02.729562 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.729276    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:02.729562 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:02.729361    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:02.729562 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:02.729455    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:02.793954 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:02.793856    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d161ea248816b49b79889a6c4d45f47" containerID="4321f64d41ca2c24e1e1f2027251d1c3914b070e9ad7c8c0da1dddfc6c51038a" exitCode=0
Sep 11 17:54:02.793954 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:02.793915    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal" event={"ID":"8d161ea248816b49b79889a6c4d45f47","Type":"ContainerDied","Data":"4321f64d41ca2c24e1e1f2027251d1c3914b070e9ad7c8c0da1dddfc6c51038a"}
Sep 11 17:54:04.728529 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:04.728492    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:04.729031 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:04.728613    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:04.729031 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:04.728960    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:04.729156 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:04.729057    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:04.729156 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:04.729109    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:04.729278 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:04.729181    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:06.334121 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:06.334070    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:06.334675 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.334218    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:06.334675 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.334288    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs podName:139a5b42-f2cc-4f37-9cac-8e7ab4ab4373 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:14.334269224 +0000 UTC m=+18.104335575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs") pod "network-metrics-daemon-hl9ct" (UID: "139a5b42-f2cc-4f37-9cac-8e7ab4ab4373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:06.435390 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:06.434857    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:06.435390 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:06.435011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:06.435390 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.435048    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:06.435390 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.435136    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret podName:e2279fb0-d7d6-42b2-b779-c9ef928edc88 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:14.435114793 +0000 UTC m=+18.205181140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret") pod "global-pull-secret-syncer-snnhr" (UID: "e2279fb0-d7d6-42b2-b779-c9ef928edc88") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:06.435390 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.435241    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:06.435390 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.435268    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:06.435390 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.435282    2571 projected.go:196] Error preparing data for projected volume kube-api-access-tvbqf for pod openshift-network-diagnostics/network-check-target-gkxjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:06.435390 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.435342    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf podName:c5ffe50c-9f38-4ba1-8c97-8a12861d6273 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:14.435324738 +0000 UTC m=+18.205391081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvbqf" (UniqueName: "kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf") pod "network-check-target-gkxjh" (UID: "c5ffe50c-9f38-4ba1-8c97-8a12861d6273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:06.725055 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:06.725016    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:06.725291 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:06.725107    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:06.725291 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:06.725154    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:06.725291 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.725163    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:06.725291 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.725257    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:06.725478 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:06.725406    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:08.724194 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:08.724056    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:08.725537 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:08.724818    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:08.725537 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:08.724897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:08.725537 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:08.724988    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:08.725537 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:08.725365    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:08.725537 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:08.725498    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:10.725141 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:10.724876    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:10.725604 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:10.724882    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:10.725604 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:10.725243    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:10.725604 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:10.724942    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:10.725604 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:10.725336    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:10.725604 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:10.725448    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:12.724752 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:12.724718    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:12.724752 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:12.724751    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:12.725250 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:12.724716    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:12.725250 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:12.724860    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:12.725250 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:12.724922    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:12.725250 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:12.724983    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:14.401112 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:14.401073    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:14.401567 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.401229    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:14.401567 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.401286    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs podName:139a5b42-f2cc-4f37-9cac-8e7ab4ab4373 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:30.401271274 +0000 UTC m=+34.171337614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs") pod "network-metrics-daemon-hl9ct" (UID: "139a5b42-f2cc-4f37-9cac-8e7ab4ab4373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:14.501933 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:14.501795    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:14.501933 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:14.501858    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:14.501933 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.501929    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:14.502214 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.501950    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:14.502214 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.501963    2571 projected.go:196] Error preparing data for projected volume kube-api-access-tvbqf for pod openshift-network-diagnostics/network-check-target-gkxjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:14.502214 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.501964    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:14.502214 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.502025    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret podName:e2279fb0-d7d6-42b2-b779-c9ef928edc88 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:30.50200845 +0000 UTC m=+34.272074798 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret") pod "global-pull-secret-syncer-snnhr" (UID: "e2279fb0-d7d6-42b2-b779-c9ef928edc88") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:14.502214 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.502044    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf podName:c5ffe50c-9f38-4ba1-8c97-8a12861d6273 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:30.50203412 +0000 UTC m=+34.272100452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvbqf" (UniqueName: "kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf") pod "network-check-target-gkxjh" (UID: "c5ffe50c-9f38-4ba1-8c97-8a12861d6273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:14.724334 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:14.724290    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:14.724525 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:14.724289    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:14.724525 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.724424    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:14.724525 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:14.724452    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:14.724671 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.724564    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:14.724671 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:14.724654    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:16.725499 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:16.725467    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:16.726001 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:16.725586    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:16.726001 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:16.725954    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:16.726114 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:16.726053    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:16.726114 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:16.726088    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:16.726209 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:16.726153    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:17.827867 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.827704    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l8kwh" event={"ID":"c3685618-7216-470e-af33-3c64fb1618c4","Type":"ContainerStarted","Data":"89dc3a1f085f4279410fd87ea30a41075e239b6a0208f51a5108bd6419ac3ff4"}
Sep 11 17:54:17.829581 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.829555    2571 generic.go:356] "Generic (PLEG): container finished" podID="9a9a9e68-6934-4724-8309-b9ef4a98ceba" containerID="ab7fc39d63359f93aee54adfec8da5f1ea4ec012f301c6ab546666eafe78524d" exitCode=0
Sep 11 17:54:17.829676 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.829636    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" event={"ID":"9a9a9e68-6934-4724-8309-b9ef4a98ceba","Type":"ContainerDied","Data":"ab7fc39d63359f93aee54adfec8da5f1ea4ec012f301c6ab546666eafe78524d"}
Sep 11 17:54:17.831691 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.831467    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zd9rf" event={"ID":"940e27d0-1102-4bb5-8b6d-22ec7cafea0c","Type":"ContainerStarted","Data":"8ba2e0806f4cda61ee7625113a9d4aa0f541237ba9839df873219cf460c75190"}
Sep 11 17:54:17.833129 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.833097    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nrr6n" event={"ID":"41a5ee9f-5818-42e5-b540-f16bbc7935a4","Type":"ContainerStarted","Data":"85ff7061deecd447124b63bd9c6058121f4c81eb83d0662d443939a54b6623ef"}
Sep 11 17:54:17.834582 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.834563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j6bg7" event={"ID":"7b5f0490-e3a5-415d-a5b5-8702db1664b3","Type":"ContainerStarted","Data":"a2029532f735c9df52dff09f8030fbb7600b69109541e2c54833e66b824a10ff"}
Sep 11 17:54:17.836260 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.836235    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79" event={"ID":"2e97aa25-b0d2-496c-b858-b38a9f3a9a51","Type":"ContainerStarted","Data":"b9eda01af1b4408ca24fbf8c4c3bf736c4f58f9abd527f6a72710ed3dc7212bc"}
Sep 11 17:54:17.840922 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.840903    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cnvwk" event={"ID":"46bb7121-35c8-46ba-a6ff-885976cea1d1","Type":"ContainerStarted","Data":"ec3a286da48f089d726511c78f0edee341bd797da573878d4152cb5440faaa22"}
Sep 11 17:54:17.842938 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.842755    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal" event={"ID":"8d161ea248816b49b79889a6c4d45f47","Type":"ContainerStarted","Data":"cc29578d912c1d4e7f9b891b81237f3f0b982a45f5f2c4ddfa92b2d842a0a45c"}
Sep 11 17:54:17.844344 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.844326    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"a9afe1c1ae8bfd05ac012e6f0ea74607e4cb156fa44728a195a370d9b09007b5"}
Sep 11 17:54:17.844419 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.844378    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"8622fd7b64f8f5d002dac676045fc40f9491a6e624f4586403659634c9601e1e"}
Sep 11 17:54:17.846152 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.846116    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l8kwh" podStartSLOduration=1.922615537 podStartE2EDuration="19.84610501s" totalImagesPullingTime="17.923489473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:17.845622012 +0000 UTC m=+21.615688361" watchObservedRunningTime="2026-09-11 17:54:17.84610501 +0000 UTC m=+21.616171358"
Sep 11 17:54:17.883607 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.883567    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zd9rf" podStartSLOduration=2.548984618 podStartE2EDuration="20.883551782s" totalImagesPullingTime="18.334567164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:17.883431699 +0000 UTC m=+21.653498047" watchObservedRunningTime="2026-09-11 17:54:17.883551782 +0000 UTC m=+21.653618130"
Sep 11 17:54:17.884136 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.884112    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cnvwk" podStartSLOduration=2.811058589 podStartE2EDuration="19.884106074s" totalImagesPullingTime="17.073047485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:17.863611967 +0000 UTC m=+21.633678325" watchObservedRunningTime="2026-09-11 17:54:17.884106074 +0000 UTC m=+21.654172422"
Sep 11 17:54:17.920867 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.920825    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-109.ec2.internal" podStartSLOduration=19.920810966 podStartE2EDuration="19.920810966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:17.920401769 +0000 UTC m=+21.690468114" watchObservedRunningTime="2026-09-11 17:54:17.920810966 +0000 UTC m=+21.690877314"
Sep 11 17:54:17.971764 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:17.971725    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j6bg7" podStartSLOduration=6.284234838 podStartE2EDuration="20.971710498s" totalImagesPullingTime="14.68747566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:17.966994195 +0000 UTC m=+21.737060542" watchObservedRunningTime="2026-09-11 17:54:17.971710498 +0000 UTC m=+21.741776845"
Sep 11 17:54:18.007712 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.007656    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nrr6n" podStartSLOduration=2.263304159 podStartE2EDuration="21.007634913s" totalImagesPullingTime="18.744330754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:18.007260461 +0000 UTC m=+21.777326811" watchObservedRunningTime="2026-09-11 17:54:18.007634913 +0000 UTC m=+21.777701262"
Sep 11 17:54:18.660970 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.660754    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 17:54:18.722089 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.721955    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:54:18.660968468Z","UUID":"6dbf9777-97f2-4869-8cff-86f77b85ce06","Handler":null,"Name":"","Endpoint":""}
Sep 11 17:54:18.723699 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.723675    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 17:54:18.723820 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.723711    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 17:54:18.727898 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.727872    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:18.728024 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.727872    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:18.728024 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:18.727973    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:18.728124 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.727872    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:18.728124 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:18.728044    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:18.728219 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:18.728173    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:18.849015 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.848978    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79" event={"ID":"2e97aa25-b0d2-496c-b858-b38a9f3a9a51","Type":"ContainerStarted","Data":"1fecd5ce4f0369f0df7c0dd1de591a8e8f255ffb26818a2a4cd80e8644e44080"}
Sep 11 17:54:18.852706 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.852675    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"241f54ae68f2cf1209ce36f7805891361b9d3dbced1811388a263b389616cd83"}
Sep 11 17:54:18.852875 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.852763    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"caf6bd70e08db8da6be839f9bbe2d6c2aed5fe40a1f3b6d912142051d48d7aac"}
Sep 11 17:54:18.852875 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.852814    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"d2cc7177b1a6b21763e53caa8d53e76481fdcfe322778d8d577ab04538a75a5c"}
Sep 11 17:54:18.852875 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.852864    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"c664cff93332152774b8c2c9cb35d4457ba8456427b8505078a5878754133b0a"}
Sep 11 17:54:18.854403 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:18.854374    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mp22s" event={"ID":"e1c8cda4-adda-44ff-a5e7-f2d7d9a341b0","Type":"ContainerStarted","Data":"a7fb8b2dff506e4e8cbb274e76970b3e63236350ed04aa25d79722779b11c755"}
Sep 11 17:54:19.815445 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:19.815361    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:54:19.815969 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:19.815952    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:54:19.837563 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:19.837519    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mp22s" podStartSLOduration=4.771015789 podStartE2EDuration="21.837502385s" totalImagesPullingTime="17.066486596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:18.871722233 +0000 UTC m=+22.641788582" watchObservedRunningTime="2026-09-11 17:54:19.837502385 +0000 UTC m=+23.607568807"
Sep 11 17:54:19.859121 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:19.859090    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79" event={"ID":"2e97aa25-b0d2-496c-b858-b38a9f3a9a51","Type":"ContainerStarted","Data":"c9edcda6bbc417aa55ed6fbc9c2bf0588316737759cc97f653787b7cdfc1e5e4"}
Sep 11 17:54:19.859572 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:19.859414    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:54:19.859687 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:19.859671    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cnvwk"
Sep 11 17:54:19.911654 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:19.911600    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tt79" podStartSLOduration=10.083639823 podStartE2EDuration="22.911581786s" totalImagesPullingTime="12.827941963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:19.884262818 +0000 UTC m=+23.654329168" watchObservedRunningTime="2026-09-11 17:54:19.911581786 +0000 UTC m=+23.681648135"
Sep 11 17:54:20.727384 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:20.727350    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:20.727564 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:20.727359    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:20.727564 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:20.727477    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:20.727668 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:20.727454    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:20.727668 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:20.727593    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:20.727766 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:20.727673    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:20.864439 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:20.864409    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"b38b622e21d056af4b52f40564e80d903b5836e05ee9d4f2e17bbc9c40da01de"}
Sep 11 17:54:22.724094 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:22.724054    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:22.724570 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:22.724063    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:22.724570 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:22.724176    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:22.724570 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:22.724192    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:22.724570 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:22.724241    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:22.724570 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:22.724332    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:24.724051 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.724014    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:24.724809 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.724060    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:24.724809 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.724026    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:24.724809 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:24.724140    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:24.724809 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:24.724218    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:24.724809 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:24.724258    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:24.873460 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.873429    2571 generic.go:356] "Generic (PLEG): container finished" podID="9a9a9e68-6934-4724-8309-b9ef4a98ceba" containerID="f8293696806cbdb4ed70317db6509be5790a9940d131d1242a7e4739a7b37512" exitCode=0
Sep 11 17:54:24.873626 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.873517    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" event={"ID":"9a9a9e68-6934-4724-8309-b9ef4a98ceba","Type":"ContainerDied","Data":"f8293696806cbdb4ed70317db6509be5790a9940d131d1242a7e4739a7b37512"}
Sep 11 17:54:24.877530 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.877506    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" event={"ID":"9323dca8-d7e0-4a88-b107-cfda2a451f8b","Type":"ContainerStarted","Data":"b5c5d032aa76909fd43a436967071d4cd1c5eb9ec3fff9f6f679d6512868dfcd"}
Sep 11 17:54:24.877761 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.877747    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:54:24.877843 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.877788    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:54:24.877843 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.877828    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:54:24.892650 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.892624    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:54:24.892779 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:24.892767    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:54:26.243544 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.243330    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h" podStartSLOduration=10.849070123 podStartE2EDuration="29.243291725s" totalImagesPullingTime="18.394221602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:24.935968779 +0000 UTC m=+28.706035127" watchObservedRunningTime="2026-09-11 17:54:26.243291725 +0000 UTC m=+30.013358074"
Sep 11 17:54:26.244078 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.244026    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-snnhr"]
Sep 11 17:54:26.244130 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.244107    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:26.244198 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:26.244181    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:26.247095 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.247074    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gkxjh"]
Sep 11 17:54:26.247202 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.247145    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:26.247248 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:26.247210    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:26.249285 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.249262    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hl9ct"]
Sep 11 17:54:26.249389 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.249376    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:26.249518 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:26.249491    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:26.884583 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.884544    2571 generic.go:356] "Generic (PLEG): container finished" podID="9a9a9e68-6934-4724-8309-b9ef4a98ceba" containerID="5cbf1ef3c67f9ed5c3e464aabb78657fb320d5a6186517f861737950ec0a6833" exitCode=0
Sep 11 17:54:26.884752 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:26.884585    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" event={"ID":"9a9a9e68-6934-4724-8309-b9ef4a98ceba","Type":"ContainerDied","Data":"5cbf1ef3c67f9ed5c3e464aabb78657fb320d5a6186517f861737950ec0a6833"}
Sep 11 17:54:27.724798 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:27.724768    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:27.725327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:27.724775    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:27.725327 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:27.724898    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:27.725327 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:27.724910    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:27.725327 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:27.724990    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:27.725327 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:27.725072    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:28.892107 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:28.892027    2571 generic.go:356] "Generic (PLEG): container finished" podID="9a9a9e68-6934-4724-8309-b9ef4a98ceba" containerID="281b6a7454e0c45baa3a3a627add194378d872c489e3681e68e298c52e7fc045" exitCode=0
Sep 11 17:54:28.892107 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:28.892080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" event={"ID":"9a9a9e68-6934-4724-8309-b9ef4a98ceba","Type":"ContainerDied","Data":"281b6a7454e0c45baa3a3a627add194378d872c489e3681e68e298c52e7fc045"}
Sep 11 17:54:29.724783 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:29.724745    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:29.724783 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:29.724770    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:29.725106 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:29.724745    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:29.725106 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:29.724863    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-snnhr" podUID="e2279fb0-d7d6-42b2-b779-c9ef928edc88"
Sep 11 17:54:29.725106 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:29.724932    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hl9ct" podUID="139a5b42-f2cc-4f37-9cac-8e7ab4ab4373"
Sep 11 17:54:29.725106 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:29.725005    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gkxjh" podUID="c5ffe50c-9f38-4ba1-8c97-8a12861d6273"
Sep 11 17:54:30.417004 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.416973    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:30.417489 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.417152    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:30.417489 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.417239    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs podName:139a5b42-f2cc-4f37-9cac-8e7ab4ab4373 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:02.417217764 +0000 UTC m=+66.187284104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs") pod "network-metrics-daemon-hl9ct" (UID: "139a5b42-f2cc-4f37-9cac-8e7ab4ab4373") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:30.518211 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.518178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:30.518411 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.518240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:30.518411 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.518357    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:30.518411 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.518374    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:30.518411 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.518394    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:30.518411 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.518408    2571 projected.go:196] Error preparing data for projected volume kube-api-access-tvbqf for pod openshift-network-diagnostics/network-check-target-gkxjh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:30.518649 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.518438    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret podName:e2279fb0-d7d6-42b2-b779-c9ef928edc88 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:02.51841628 +0000 UTC m=+66.288482624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret") pod "global-pull-secret-syncer-snnhr" (UID: "e2279fb0-d7d6-42b2-b779-c9ef928edc88") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:30.518649 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.518459    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf podName:c5ffe50c-9f38-4ba1-8c97-8a12861d6273 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:02.51844508 +0000 UTC m=+66.288511422 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvbqf" (UniqueName: "kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf") pod "network-check-target-gkxjh" (UID: "c5ffe50c-9f38-4ba1-8c97-8a12861d6273") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:30.556068 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.556040    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-109.ec2.internal" event="NodeReady"
Sep 11 17:54:30.556220 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.556169    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 17:54:30.601947 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.601905    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b899cf4d4-r5nv9"]
Sep 11 17:54:30.672732 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.672479    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b899cf4d4-r5nv9"]
Sep 11 17:54:30.672886 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.672758    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lgnjx"]
Sep 11 17:54:30.672886 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.672619    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.677025 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.677000    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 17:54:30.677574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.677554    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 17:54:30.677968 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.677642    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 17:54:30.677968 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.677696    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5487c\""
Sep 11 17:54:30.684515 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.684494    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hn9nt"]
Sep 11 17:54:30.684619 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.684600    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.685862 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.685845    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 17:54:30.688384 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.688361    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cj7w5\""
Sep 11 17:54:30.688643 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.688627    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 17:54:30.689175 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.689148    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 17:54:30.702737 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.702711    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lgnjx"]
Sep 11 17:54:30.702842 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.702751    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hn9nt"]
Sep 11 17:54:30.702902 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.702850    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hn9nt"
Sep 11 17:54:30.705813 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.705793    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 17:54:30.705945 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.705927    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7qbb\""
Sep 11 17:54:30.706158 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.706140    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 17:54:30.706249 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.706211    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 17:54:30.819848 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.819822    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.819995 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.819854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-ca-trust-extracted\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.819995 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.819878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwxr\" (UniqueName: \"kubernetes.io/projected/e860867b-f1e8-4d20-a11b-bb01c144fdc2-kube-api-access-hzwxr\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.819995 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.819936    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-trusted-ca\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.819995 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.819992    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-installation-pull-secrets\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.820127 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820024    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-kube-api-access-td49m\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.820127 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820062    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.820127 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-certificates\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.820127 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/065a0aab-56f6-461e-bfb1-e9a51a830677-cert\") pod \"ingress-canary-hn9nt\" (UID: \"065a0aab-56f6-461e-bfb1-e9a51a830677\") " pod="openshift-ingress-canary/ingress-canary-hn9nt"
Sep 11 17:54:30.820243 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820154    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrrs\" (UniqueName: \"kubernetes.io/projected/065a0aab-56f6-461e-bfb1-e9a51a830677-kube-api-access-ftrrs\") pod \"ingress-canary-hn9nt\" (UID: \"065a0aab-56f6-461e-bfb1-e9a51a830677\") " pod="openshift-ingress-canary/ingress-canary-hn9nt"
Sep 11 17:54:30.820243 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820172    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e860867b-f1e8-4d20-a11b-bb01c144fdc2-tmp-dir\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.820243 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820209    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-image-registry-private-configuration\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.820243 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820235    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-bound-sa-token\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.820381 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.820248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e860867b-f1e8-4d20-a11b-bb01c144fdc2-config-volume\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.830785 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.830757    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:30.921431 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrrs\" (UniqueName: \"kubernetes.io/projected/065a0aab-56f6-461e-bfb1-e9a51a830677-kube-api-access-ftrrs\") pod \"ingress-canary-hn9nt\" (UID: \"065a0aab-56f6-461e-bfb1-e9a51a830677\") " pod="openshift-ingress-canary/ingress-canary-hn9nt"
Sep 11 17:54:30.921431 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921436    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e860867b-f1e8-4d20-a11b-bb01c144fdc2-tmp-dir\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.921641 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921462    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-image-registry-private-configuration\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.921641 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921480    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-bound-sa-token\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.921641 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e860867b-f1e8-4d20-a11b-bb01c144fdc2-config-volume\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.921641 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921517    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.921641 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-ca-trust-extracted\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.921641 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921569    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwxr\" (UniqueName: \"kubernetes.io/projected/e860867b-f1e8-4d20-a11b-bb01c144fdc2-kube-api-access-hzwxr\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.921641 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921602    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-trusted-ca\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.921931 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.921753    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:30.921931 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.921835    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls podName:e860867b-f1e8-4d20-a11b-bb01c144fdc2 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:31.421812105 +0000 UTC m=+35.191878435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls") pod "dns-default-lgnjx" (UID: "e860867b-f1e8-4d20-a11b-bb01c144fdc2") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:30.921931 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921884    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-installation-pull-secrets\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.921931 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-kube-api-access-td49m\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.922137 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921944    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.922137 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921969    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-certificates\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.922137 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921969    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e860867b-f1e8-4d20-a11b-bb01c144fdc2-tmp-dir\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.922137 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.921999    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/065a0aab-56f6-461e-bfb1-e9a51a830677-cert\") pod \"ingress-canary-hn9nt\" (UID: \"065a0aab-56f6-461e-bfb1-e9a51a830677\") " pod="openshift-ingress-canary/ingress-canary-hn9nt"
Sep 11 17:54:30.922137 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.922045    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e860867b-f1e8-4d20-a11b-bb01c144fdc2-config-volume\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.922364 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.922173    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:54:30.922364 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.922183    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b899cf4d4-r5nv9: secret "image-registry-tls" not found
Sep 11 17:54:30.922364 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:30.922226    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls podName:cbdc3961-c5bf-48f8-b914-ca7a2981a9d7 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:31.422215429 +0000 UTC m=+35.192281758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls") pod "image-registry-5b899cf4d4-r5nv9" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7") : secret "image-registry-tls" not found
Sep 11 17:54:30.922872 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.922558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-ca-trust-extracted\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.922872 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.922802    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-certificates\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.922994 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.922976    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-trusted-ca\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.925579 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.925557    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-image-registry-private-configuration\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.925579 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.925578    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/065a0aab-56f6-461e-bfb1-e9a51a830677-cert\") pod \"ingress-canary-hn9nt\" (UID: \"065a0aab-56f6-461e-bfb1-e9a51a830677\") " pod="openshift-ingress-canary/ingress-canary-hn9nt"
Sep 11 17:54:30.925704 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.925607    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-installation-pull-secrets\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.943721 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.943697    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-bound-sa-token\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.943863 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.943828    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrrs\" (UniqueName: \"kubernetes.io/projected/065a0aab-56f6-461e-bfb1-e9a51a830677-kube-api-access-ftrrs\") pod \"ingress-canary-hn9nt\" (UID: \"065a0aab-56f6-461e-bfb1-e9a51a830677\") " pod="openshift-ingress-canary/ingress-canary-hn9nt"
Sep 11 17:54:30.944514 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.944496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwxr\" (UniqueName: \"kubernetes.io/projected/e860867b-f1e8-4d20-a11b-bb01c144fdc2-kube-api-access-hzwxr\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:30.945283 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.945257    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-kube-api-access-td49m\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:30.950736 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:30.950715    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:31.013581 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.013550    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hn9nt"
Sep 11 17:54:31.354198 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.353899    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hn9nt"]
Sep 11 17:54:31.356712 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:54:31.356682    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065a0aab_56f6_461e_bfb1_e9a51a830677.slice/crio-36b0f8fa817f97d362c193e028493e9c5c7f2d83ad49fe1947999de4bd169445 WatchSource:0}: Error finding container 36b0f8fa817f97d362c193e028493e9c5c7f2d83ad49fe1947999de4bd169445: Status 404 returned error can't find the container with id 36b0f8fa817f97d362c193e028493e9c5c7f2d83ad49fe1947999de4bd169445
Sep 11 17:54:31.425036 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.424998    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:31.425753 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.425077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:31.425753 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:31.425189    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:31.425753 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:31.425266    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls podName:e860867b-f1e8-4d20-a11b-bb01c144fdc2 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:32.425242494 +0000 UTC m=+36.195308840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls") pod "dns-default-lgnjx" (UID: "e860867b-f1e8-4d20-a11b-bb01c144fdc2") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:31.425753 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:31.425192    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:54:31.425753 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:31.425391    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b899cf4d4-r5nv9: secret "image-registry-tls" not found
Sep 11 17:54:31.425753 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:31.425456    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls podName:cbdc3961-c5bf-48f8-b914-ca7a2981a9d7 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:32.425436146 +0000 UTC m=+36.195502481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls") pod "image-registry-5b899cf4d4-r5nv9" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7") : secret "image-registry-tls" not found
Sep 11 17:54:31.724734 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.724703    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:54:31.724929 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.724811    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:54:31.724929 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.724838    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:54:31.731599 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.731444    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:54:31.731599 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.731504    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:54:31.732577 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.732409    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:54:31.732577 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.732419    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhttf\""
Sep 11 17:54:31.732856 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.732836    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:54:31.732961 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.732836    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-64859\""
Sep 11 17:54:31.900498 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:31.900465    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hn9nt" event={"ID":"065a0aab-56f6-461e-bfb1-e9a51a830677","Type":"ContainerStarted","Data":"36b0f8fa817f97d362c193e028493e9c5c7f2d83ad49fe1947999de4bd169445"}
Sep 11 17:54:32.433587 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:32.433547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:32.434116 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:32.433726    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:54:32.434116 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:32.433750    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b899cf4d4-r5nv9: secret "image-registry-tls" not found
Sep 11 17:54:32.434116 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:32.433728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:32.434116 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:32.433817    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls podName:cbdc3961-c5bf-48f8-b914-ca7a2981a9d7 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:34.433795526 +0000 UTC m=+38.203861856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls") pod "image-registry-5b899cf4d4-r5nv9" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7") : secret "image-registry-tls" not found
Sep 11 17:54:32.434116 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:32.433818    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:32.434116 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:32.433868    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls podName:e860867b-f1e8-4d20-a11b-bb01c144fdc2 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:34.43385445 +0000 UTC m=+38.203920797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls") pod "dns-default-lgnjx" (UID: "e860867b-f1e8-4d20-a11b-bb01c144fdc2") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:34.449549 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:34.449493    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:34.450017 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:34.449570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:34.450017 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:34.449671    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:54:34.450017 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:34.449688    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b899cf4d4-r5nv9: secret "image-registry-tls" not found
Sep 11 17:54:34.450017 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:34.449667    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:34.450017 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:34.449757    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls podName:e860867b-f1e8-4d20-a11b-bb01c144fdc2 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:38.449739152 +0000 UTC m=+42.219805492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls") pod "dns-default-lgnjx" (UID: "e860867b-f1e8-4d20-a11b-bb01c144fdc2") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:34.450017 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:34.449770    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls podName:cbdc3961-c5bf-48f8-b914-ca7a2981a9d7 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:38.449763832 +0000 UTC m=+42.219830159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls") pod "image-registry-5b899cf4d4-r5nv9" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7") : secret "image-registry-tls" not found
Sep 11 17:54:35.714789 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:35.714601    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:35.812564 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:35.812528    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:35.911864 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:35.911828    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hn9nt" event={"ID":"065a0aab-56f6-461e-bfb1-e9a51a830677","Type":"ContainerStarted","Data":"99e90616bb2ef6b6046625379dc0f9188f49b4277e7f077bb9280ca286a18f3b"}
Sep 11 17:54:35.914521 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:35.914495    2571 generic.go:356] "Generic (PLEG): container finished" podID="9a9a9e68-6934-4724-8309-b9ef4a98ceba" containerID="5fe397de9ea7000492da15e20fa87a005eb51f65b303e69bdb1f3c5efaa9aa9a" exitCode=0
Sep 11 17:54:35.914631 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:35.914549    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" event={"ID":"9a9a9e68-6934-4724-8309-b9ef4a98ceba","Type":"ContainerDied","Data":"5fe397de9ea7000492da15e20fa87a005eb51f65b303e69bdb1f3c5efaa9aa9a"}
Sep 11 17:54:35.986972 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:35.986928    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hn9nt" podStartSLOduration=1.9642854060000001 podStartE2EDuration="5.986903385s" totalImagesPullingTime="4.022617979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:35.951279669 +0000 UTC m=+39.721346017" watchObservedRunningTime="2026-09-11 17:54:35.986903385 +0000 UTC m=+39.756969734"
Sep 11 17:54:36.919830 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:36.919792    2571 generic.go:356] "Generic (PLEG): container finished" podID="9a9a9e68-6934-4724-8309-b9ef4a98ceba" containerID="2619bb5b0612d85c1adb7bb9afe48713c81cf3516d01cc72ae4be80c9bbf2926" exitCode=0
Sep 11 17:54:36.920183 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:36.919877    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" event={"ID":"9a9a9e68-6934-4724-8309-b9ef4a98ceba","Type":"ContainerDied","Data":"2619bb5b0612d85c1adb7bb9afe48713c81cf3516d01cc72ae4be80c9bbf2926"}
Sep 11 17:54:37.116567 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:37.116533    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:37.195447 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:37.195372    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:37.926027 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:37.925993    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" event={"ID":"9a9a9e68-6934-4724-8309-b9ef4a98ceba","Type":"ContainerStarted","Data":"84c277978a894a71aa9c5b01fd9c557a351770c3f54a96ec5d6f50f7c71824e9"}
Sep 11 17:54:37.952564 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:37.952517    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jmsqt" podStartSLOduration=11.59388323 podStartE2EDuration="40.952502098s" totalImagesPullingTime="29.358618868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:37.951000662 +0000 UTC m=+41.721067021" watchObservedRunningTime="2026-09-11 17:54:37.952502098 +0000 UTC m=+41.722568445"
Sep 11 17:54:38.085690 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:38.085647    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:38.186760 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:38.186691    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log" limit=1024
Sep 11 17:54:38.480056 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:38.479976    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:38.480056 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:38.480027    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:38.480216 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:38.480109    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:38.480216 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:38.480114    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:54:38.480216 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:38.480129    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b899cf4d4-r5nv9: secret "image-registry-tls" not found
Sep 11 17:54:38.480216 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:38.480161    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls podName:e860867b-f1e8-4d20-a11b-bb01c144fdc2 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:46.480147953 +0000 UTC m=+50.250214299 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls") pod "dns-default-lgnjx" (UID: "e860867b-f1e8-4d20-a11b-bb01c144fdc2") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:38.480216 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:38.480176    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls podName:cbdc3961-c5bf-48f8-b914-ca7a2981a9d7 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:46.480170158 +0000 UTC m=+50.250236484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls") pod "image-registry-5b899cf4d4-r5nv9" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7") : secret "image-registry-tls" not found
Sep 11 17:54:46.540155 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:46.540116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:54:46.540672 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:46.540186    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:54:46.540672 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:46.540248    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:54:46.540672 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:46.540272    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b899cf4d4-r5nv9: secret "image-registry-tls" not found
Sep 11 17:54:46.540672 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:46.540313    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:46.540672 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:46.540384    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls podName:cbdc3961-c5bf-48f8-b914-ca7a2981a9d7 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:02.540368537 +0000 UTC m=+66.310434876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls") pod "image-registry-5b899cf4d4-r5nv9" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7") : secret "image-registry-tls" not found
Sep 11 17:54:46.540672 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:46.540397    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls podName:e860867b-f1e8-4d20-a11b-bb01c144fdc2 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:02.540391977 +0000 UTC m=+66.310458303 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls") pod "dns-default-lgnjx" (UID: "e860867b-f1e8-4d20-a11b-bb01c144fdc2") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:56.895264 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:56.895237    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4v47h"
Sep 11 17:54:57.665551 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.665513    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nnjnk"]
Sep 11 17:54:57.668842 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.668826    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.671595 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.671575    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 17:54:57.671766 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.671577    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 17:54:57.671766 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.671642    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntvnk\""
Sep 11 17:54:57.671766 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.671699    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 17:54:57.672662 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.672648    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 17:54:57.689075 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.689052    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nnjnk"]
Sep 11 17:54:57.718170 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.718137    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.718351 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.718210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhpr\" (UniqueName: \"kubernetes.io/projected/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-api-access-bwhpr\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.718351 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.718254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed74f2d5-3e16-4910-8e84-12da8638b9a6-data-volume\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.718351 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.718273    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed74f2d5-3e16-4910-8e84-12da8638b9a6-crio-socket\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.718351 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.718312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.818795 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.818742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhpr\" (UniqueName: \"kubernetes.io/projected/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-api-access-bwhpr\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.818908 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.818827    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed74f2d5-3e16-4910-8e84-12da8638b9a6-data-volume\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.818961 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.818914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed74f2d5-3e16-4910-8e84-12da8638b9a6-crio-socket\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.818961 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.818930    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.818961 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.818951    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.819115 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.819003    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ed74f2d5-3e16-4910-8e84-12da8638b9a6-crio-socket\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.819115 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:57.819042    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:54:57.819115 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:57.819042    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:57.819261 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:57.819110    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:58.31909558 +0000 UTC m=+62.089161925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : configmap "kube-rbac-proxy" not found
Sep 11 17:54:57.819261 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:57.819175    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:58.319161605 +0000 UTC m=+62.089227931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:57.819261 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.819190    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed74f2d5-3e16-4910-8e84-12da8638b9a6-data-volume\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:57.830910 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:57.830886    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhpr\" (UniqueName: \"kubernetes.io/projected/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-api-access-bwhpr\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:58.323097 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:58.323063    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:58.323097 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:58.323104    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:58.323611 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:58.323200    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:54:58.323611 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:58.323211    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:58.323611 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:58.323260    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:59.323245492 +0000 UTC m=+63.093311834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : configmap "kube-rbac-proxy" not found
Sep 11 17:54:58.323611 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:58.323274    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:59.323268662 +0000 UTC m=+63.093335001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:59.329941 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:59.329906    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:59.330408 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:59.329952    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:54:59.330408 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:59.330032    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:54:59.330408 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:59.330072    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:59.330408 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:59.330106    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.330089316 +0000 UTC m=+65.100155642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : configmap "kube-rbac-proxy" not found
Sep 11 17:54:59.330408 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:54:59.330148    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.33012991 +0000 UTC m=+65.100196236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:59.412525 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:54:59.412504    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l8kwh_c3685618-7216-470e-af33-3c64fb1618c4/dns-node-resolver/0.log"
Sep 11 17:55:00.412214 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:00.412189    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j6bg7_7b5f0490-e3a5-415d-a5b5-8702db1664b3/node-ca/0.log"
Sep 11 17:55:01.346456 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:01.346405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:01.346456 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:01.346457    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:01.346666 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:01.346552    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:01.346666 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:01.346584    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:55:01.346666 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:01.346616    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:05.346600961 +0000 UTC m=+69.116667286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:01.346666 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:01.346641    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:05.346621914 +0000 UTC m=+69.116688241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : configmap "kube-rbac-proxy" not found
Sep 11 17:55:01.412853 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:01.412828    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hn9nt_065a0aab-56f6-461e-bfb1-e9a51a830677/serve-healthcheck-canary/0.log"
Sep 11 17:55:02.453979 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.453935    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:55:02.456928 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.456913    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:55:02.464670 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:02.464654    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 17:55:02.464722 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:02.464704    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs podName:139a5b42-f2cc-4f37-9cac-8e7ab4ab4373 nodeName:}" failed. No retries permitted until 2026-09-11 17:56:06.46468999 +0000 UTC m=+130.234756317 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs") pod "network-metrics-daemon-hl9ct" (UID: "139a5b42-f2cc-4f37-9cac-8e7ab4ab4373") : secret "metrics-daemon-secret" not found
Sep 11 17:55:02.555080 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.555055    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:55:02.555151 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.555126    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:55:02.555186 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.555155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:55:02.555186 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.555181    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:55:02.555255 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:02.555200    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:55:02.555255 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:02.555251    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls podName:e860867b-f1e8-4d20-a11b-bb01c144fdc2 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:34.555237965 +0000 UTC m=+98.325304291 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls") pod "dns-default-lgnjx" (UID: "e860867b-f1e8-4d20-a11b-bb01c144fdc2") : secret "dns-default-metrics-tls" not found
Sep 11 17:55:02.555363 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:02.555281    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 17:55:02.555363 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:02.555323    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b899cf4d4-r5nv9: secret "image-registry-tls" not found
Sep 11 17:55:02.555427 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:02.555369    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls podName:cbdc3961-c5bf-48f8-b914-ca7a2981a9d7 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:34.55535439 +0000 UTC m=+98.325420720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls") pod "image-registry-5b899cf4d4-r5nv9" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7") : secret "image-registry-tls" not found
Sep 11 17:55:02.563338 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.563317    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:55:02.564918 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.564903    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:55:02.568009 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.567976    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2279fb0-d7d6-42b2-b779-c9ef928edc88-original-pull-secret\") pod \"global-pull-secret-syncer-snnhr\" (UID: \"e2279fb0-d7d6-42b2-b779-c9ef928edc88\") " pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:55:02.575942 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.575920    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:55:02.578118 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.578098    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbqf\" (UniqueName: \"kubernetes.io/projected/c5ffe50c-9f38-4ba1-8c97-8a12861d6273-kube-api-access-tvbqf\") pod \"network-check-target-gkxjh\" (UID: \"c5ffe50c-9f38-4ba1-8c97-8a12861d6273\") " pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:55:02.642558 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.642536    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhttf\""
Sep 11 17:55:02.646094 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.646078    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-snnhr"
Sep 11 17:55:02.649204 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.649187    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:55:02.929360 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.929322    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-snnhr"]
Sep 11 17:55:02.930043 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.930021    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gkxjh"]
Sep 11 17:55:02.932882 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:02.932856    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ffe50c_9f38_4ba1_8c97_8a12861d6273.slice/crio-9ab9ec14efd3b5aac3576f459847cf0f855c7016dca0f15d194988623fdbabca WatchSource:0}: Error finding container 9ab9ec14efd3b5aac3576f459847cf0f855c7016dca0f15d194988623fdbabca: Status 404 returned error can't find the container with id 9ab9ec14efd3b5aac3576f459847cf0f855c7016dca0f15d194988623fdbabca
Sep 11 17:55:02.933332 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:02.933306    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2279fb0_d7d6_42b2_b779_c9ef928edc88.slice/crio-93489c4d19228ac608c8fa107e0a0592da45193d9821eca9391c9c6a1f2b74b4 WatchSource:0}: Error finding container 93489c4d19228ac608c8fa107e0a0592da45193d9821eca9391c9c6a1f2b74b4: Status 404 returned error can't find the container with id 93489c4d19228ac608c8fa107e0a0592da45193d9821eca9391c9c6a1f2b74b4
Sep 11 17:55:02.983622 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.983591    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-snnhr" event={"ID":"e2279fb0-d7d6-42b2-b779-c9ef928edc88","Type":"ContainerStarted","Data":"93489c4d19228ac608c8fa107e0a0592da45193d9821eca9391c9c6a1f2b74b4"}
Sep 11 17:55:02.984666 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:02.984644    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gkxjh" event={"ID":"c5ffe50c-9f38-4ba1-8c97-8a12861d6273","Type":"ContainerStarted","Data":"9ab9ec14efd3b5aac3576f459847cf0f855c7016dca0f15d194988623fdbabca"}
Sep 11 17:55:03.904413 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.904359    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"]
Sep 11 17:55:03.907573 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.907548    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:03.910336 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.910313    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 11 17:55:03.910463 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.910353    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 11 17:55:03.911455 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.911433    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gt8wp\""
Sep 11 17:55:03.911867 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.911738    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 11 17:55:03.911867 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.911854    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:55:03.918054 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.918025    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"]
Sep 11 17:55:03.970564 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.970478    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14a35285-829b-490a-a879-8b415812e26a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:03.970564 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.970525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14a35285-829b-490a-a879-8b415812e26a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:03.970792 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:03.970584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9hgv\" (UniqueName: \"kubernetes.io/projected/14a35285-829b-490a-a879-8b415812e26a-kube-api-access-l9hgv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:04.071314 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.071270    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14a35285-829b-490a-a879-8b415812e26a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:04.071496 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.071325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14a35285-829b-490a-a879-8b415812e26a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:04.071496 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.071387    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9hgv\" (UniqueName: \"kubernetes.io/projected/14a35285-829b-490a-a879-8b415812e26a-kube-api-access-l9hgv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:04.072087 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.072027    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/14a35285-829b-490a-a879-8b415812e26a-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:04.074276 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.074244    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/14a35285-829b-490a-a879-8b415812e26a-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:04.082156 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.082119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9hgv\" (UniqueName: \"kubernetes.io/projected/14a35285-829b-490a-a879-8b415812e26a-kube-api-access-l9hgv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-bzsls\" (UID: \"14a35285-829b-490a-a879-8b415812e26a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:04.219685 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.219612    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"
Sep 11 17:55:04.496745 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.496670    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls"]
Sep 11 17:55:04.500696 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:04.500666    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a35285_829b_490a_a879_8b415812e26a.slice/crio-b69151981f23781e70b91979527405aebfbf36f4cea72786323ac10ff707b671 WatchSource:0}: Error finding container b69151981f23781e70b91979527405aebfbf36f4cea72786323ac10ff707b671: Status 404 returned error can't find the container with id b69151981f23781e70b91979527405aebfbf36f4cea72786323ac10ff707b671
Sep 11 17:55:04.992870 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:04.992822    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls" event={"ID":"14a35285-829b-490a-a879-8b415812e26a","Type":"ContainerStarted","Data":"b69151981f23781e70b91979527405aebfbf36f4cea72786323ac10ff707b671"}
Sep 11 17:55:05.381897 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:05.381808    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:05.381897 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:05.381873    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:05.382124 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:05.381975    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:05.382124 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:05.382012    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:55:05.382124 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:05.382047    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:13.382028353 +0000 UTC m=+77.152094693 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:05.382124 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:05.382074    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:13.382055889 +0000 UTC m=+77.152122229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : configmap "kube-rbac-proxy" not found
Sep 11 17:55:07.476393 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:07.476355    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fzzll"]
Sep 11 17:55:07.492891 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:07.492851    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fzzll"]
Sep 11 17:55:07.493043 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:07.492968    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fzzll"
Sep 11 17:55:07.495738 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:07.495718    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-57wgs\""
Sep 11 17:55:07.599184 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:07.599150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrpph\" (UniqueName: \"kubernetes.io/projected/51124087-9fdb-4f01-bb66-7d6a7bbcee50-kube-api-access-wrpph\") pod \"network-check-source-6f8b768455-fzzll\" (UID: \"51124087-9fdb-4f01-bb66-7d6a7bbcee50\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fzzll"
Sep 11 17:55:07.703234 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:07.700616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrpph\" (UniqueName: \"kubernetes.io/projected/51124087-9fdb-4f01-bb66-7d6a7bbcee50-kube-api-access-wrpph\") pod \"network-check-source-6f8b768455-fzzll\" (UID: \"51124087-9fdb-4f01-bb66-7d6a7bbcee50\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fzzll"
Sep 11 17:55:07.710169 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:07.710141    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrpph\" (UniqueName: \"kubernetes.io/projected/51124087-9fdb-4f01-bb66-7d6a7bbcee50-kube-api-access-wrpph\") pod \"network-check-source-6f8b768455-fzzll\" (UID: \"51124087-9fdb-4f01-bb66-7d6a7bbcee50\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-fzzll"
Sep 11 17:55:07.802974 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:07.802900    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fzzll"
Sep 11 17:55:08.989484 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:08.989430    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-fzzll"]
Sep 11 17:55:08.993014 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:08.992981    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51124087_9fdb_4f01_bb66_7d6a7bbcee50.slice/crio-019a3f0d914a8fd055dba186ea8f31d14062f6b1637909988e177b3aa1d1e485 WatchSource:0}: Error finding container 019a3f0d914a8fd055dba186ea8f31d14062f6b1637909988e177b3aa1d1e485: Status 404 returned error can't find the container with id 019a3f0d914a8fd055dba186ea8f31d14062f6b1637909988e177b3aa1d1e485
Sep 11 17:55:09.005434 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:09.005404    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-snnhr" event={"ID":"e2279fb0-d7d6-42b2-b779-c9ef928edc88","Type":"ContainerStarted","Data":"eaf168ab5bc09c957bb4dc157605184ec40a8fced7c24d926bb961364acf7886"}
Sep 11 17:55:09.006894 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:09.006871    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gkxjh" event={"ID":"c5ffe50c-9f38-4ba1-8c97-8a12861d6273","Type":"ContainerStarted","Data":"3c511eabd46c39d16fea5b56dce7783e18cb84e6a3f82cb235498aae592733cf"}
Sep 11 17:55:09.008352 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:09.008329    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls" event={"ID":"14a35285-829b-490a-a879-8b415812e26a","Type":"ContainerStarted","Data":"5cec127a1af01b44fbf9da5a32e2b0c100515914881d036534740c0ff75f5e7e"}
Sep 11 17:55:09.009482 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:09.009450    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fzzll" event={"ID":"51124087-9fdb-4f01-bb66-7d6a7bbcee50","Type":"ContainerStarted","Data":"019a3f0d914a8fd055dba186ea8f31d14062f6b1637909988e177b3aa1d1e485"}
Sep 11 17:55:09.037976 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:09.037875    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-snnhr" podStartSLOduration=65.246902176 podStartE2EDuration="1m11.03785676s" totalImagesPullingTime="5.790954584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:09.037343929 +0000 UTC m=+72.807410278" watchObservedRunningTime="2026-09-11 17:55:09.03785676 +0000 UTC m=+72.807923108"
Sep 11 17:55:09.056591 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:09.056550    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gkxjh" podStartSLOduration=66.26469711 podStartE2EDuration="1m12.056534408s" totalImagesPullingTime="5.791837298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:09.055456739 +0000 UTC m=+72.825523087" watchObservedRunningTime="2026-09-11 17:55:09.056534408 +0000 UTC m=+72.826600752"
Sep 11 17:55:09.075601 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:09.075546    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls" podStartSLOduration=1.846342559 podStartE2EDuration="6.075523524s" totalImagesPullingTime="4.229180965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:09.074696064 +0000 UTC m=+72.844762413" watchObservedRunningTime="2026-09-11 17:55:09.075523524 +0000 UTC m=+72.845589873"
Sep 11 17:55:10.014228 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.014190    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fzzll" event={"ID":"51124087-9fdb-4f01-bb66-7d6a7bbcee50","Type":"ContainerStarted","Data":"67744701a0c815da9fff274cf84de04e84399538370c1a97991d36b93ec407d3"}
Sep 11 17:55:10.014713 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.014279    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:55:10.034714 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.034666    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-fzzll" podStartSLOduration=3.034650267 podStartE2EDuration="3.034650267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:10.032895813 +0000 UTC m=+73.802962161" watchObservedRunningTime="2026-09-11 17:55:10.034650267 +0000 UTC m=+73.804716614"
Sep 11 17:55:10.534978 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.534940    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp"]
Sep 11 17:55:10.536930 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.536914    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp"
Sep 11 17:55:10.541862 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.541836    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 11 17:55:10.542133 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.542112    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 11 17:55:10.543117 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.543098    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-l8vtg\""
Sep 11 17:55:10.550663 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.550643    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp"]
Sep 11 17:55:10.621959 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.621924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9glft\" (UniqueName: \"kubernetes.io/projected/dd90b17f-50cf-480d-951d-42bf44b98377-kube-api-access-9glft\") pod \"migrator-8b4d697c5-m8gqp\" (UID: \"dd90b17f-50cf-480d-951d-42bf44b98377\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp"
Sep 11 17:55:10.723123 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.723089    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9glft\" (UniqueName: \"kubernetes.io/projected/dd90b17f-50cf-480d-951d-42bf44b98377-kube-api-access-9glft\") pod \"migrator-8b4d697c5-m8gqp\" (UID: \"dd90b17f-50cf-480d-951d-42bf44b98377\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp"
Sep 11 17:55:10.733431 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.733402    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9glft\" (UniqueName: \"kubernetes.io/projected/dd90b17f-50cf-480d-951d-42bf44b98377-kube-api-access-9glft\") pod \"migrator-8b4d697c5-m8gqp\" (UID: \"dd90b17f-50cf-480d-951d-42bf44b98377\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp"
Sep 11 17:55:10.845921 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:10.845829    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp"
Sep 11 17:55:11.109413 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:11.109327    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp"]
Sep 11 17:55:11.112081 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:11.112051    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd90b17f_50cf_480d_951d_42bf44b98377.slice/crio-57ac60375770ced04dd58dea106fe3044cc441b243e8ab8d808f17889c008deb WatchSource:0}: Error finding container 57ac60375770ced04dd58dea106fe3044cc441b243e8ab8d808f17889c008deb: Status 404 returned error can't find the container with id 57ac60375770ced04dd58dea106fe3044cc441b243e8ab8d808f17889c008deb
Sep 11 17:55:12.020934 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:12.020898    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp" event={"ID":"dd90b17f-50cf-480d-951d-42bf44b98377","Type":"ContainerStarted","Data":"57ac60375770ced04dd58dea106fe3044cc441b243e8ab8d808f17889c008deb"}
Sep 11 17:55:13.025343 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:13.025311    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp" event={"ID":"dd90b17f-50cf-480d-951d-42bf44b98377","Type":"ContainerStarted","Data":"0ac42baffa48e2974dd9c63bcafb568264a9cb92969796e4c52af65a5f3a98b8"}
Sep 11 17:55:13.025725 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:13.025364    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp" event={"ID":"dd90b17f-50cf-480d-951d-42bf44b98377","Type":"ContainerStarted","Data":"4515f5123c9d5d7cb35d6f73b251bd1d81749ac0f9e0937c94ac6fe2b790cb60"}
Sep 11 17:55:13.049142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:13.049091    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-m8gqp" podStartSLOduration=1.839793115 podStartE2EDuration="3.049074969s" totalImagesPullingTime="1.209281854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:13.047483774 +0000 UTC m=+76.817550122" watchObservedRunningTime="2026-09-11 17:55:13.049074969 +0000 UTC m=+76.819141316"
Sep 11 17:55:13.444209 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:13.444171    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:13.444413 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:13.444221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:13.444413 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:13.444341    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:13.444413 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:55:13.444407    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls podName:ed74f2d5-3e16-4910-8e84-12da8638b9a6 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:29.44439224 +0000 UTC m=+93.214458566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nnjnk" (UID: "ed74f2d5-3e16-4910-8e84-12da8638b9a6") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:13.444759 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:13.444741    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ed74f2d5-3e16-4910-8e84-12da8638b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:29.456337 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:29.456176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:29.458465 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:29.458445    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ed74f2d5-3e16-4910-8e84-12da8638b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nnjnk\" (UID: \"ed74f2d5-3e16-4910-8e84-12da8638b9a6\") " pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:29.477410 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:29.477383    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nnjnk"
Sep 11 17:55:29.745906 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:29.745884    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nnjnk"]
Sep 11 17:55:29.748453 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:29.748427    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded74f2d5_3e16_4910_8e84_12da8638b9a6.slice/crio-28e76885efafb9e6026038df656a33b8d17ef577f75d83f07b375e6b0cd238a8 WatchSource:0}: Error finding container 28e76885efafb9e6026038df656a33b8d17ef577f75d83f07b375e6b0cd238a8: Status 404 returned error can't find the container with id 28e76885efafb9e6026038df656a33b8d17ef577f75d83f07b375e6b0cd238a8
Sep 11 17:55:30.085896 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:30.085807    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nnjnk" event={"ID":"ed74f2d5-3e16-4910-8e84-12da8638b9a6","Type":"ContainerStarted","Data":"7028a0087222a66b682490d67f10490a6d3915fb79cdffe357ba689dc100d90a"}
Sep 11 17:55:30.085896 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:30.085870    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nnjnk" event={"ID":"ed74f2d5-3e16-4910-8e84-12da8638b9a6","Type":"ContainerStarted","Data":"28e76885efafb9e6026038df656a33b8d17ef577f75d83f07b375e6b0cd238a8"}
Sep 11 17:55:31.090784 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:31.090744    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nnjnk" event={"ID":"ed74f2d5-3e16-4910-8e84-12da8638b9a6","Type":"ContainerStarted","Data":"dfa3c860aee52d246f88b498af421e223746a559e70721fed0feaa69961bdd9e"}
Sep 11 17:55:32.097937 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:32.097855    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nnjnk" event={"ID":"ed74f2d5-3e16-4910-8e84-12da8638b9a6","Type":"ContainerStarted","Data":"986418d563086926f2d127218c37a3ef3c95b3a7d0e1cb6fa781b0667600a0cb"}
Sep 11 17:55:32.122261 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:32.122211    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nnjnk" podStartSLOduration=33.150513391 podStartE2EDuration="35.122193403s" totalImagesPullingTime="1.971680012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:32.121843317 +0000 UTC m=+95.891909666" watchObservedRunningTime="2026-09-11 17:55:32.122193403 +0000 UTC m=+95.892259751"
Sep 11 17:55:34.483569 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.483535    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"]
Sep 11 17:55:34.486686 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.486670    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.489727 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.489700    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 17:55:34.489847 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.489739    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 17:55:34.490008 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.489992    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 17:55:34.490046 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.489994    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 17:55:34.491231 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.491214    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 17:55:34.491231 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.491224    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 17:55:34.491367 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.491347    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 17:55:34.504896 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.504874    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"]
Sep 11 17:55:34.593902 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.593865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:55:34.594073 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.593911    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/264c2b17-f41b-4127-9c1d-84c956d9706f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.594073 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.593944    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-hub\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.594073 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.593968    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.594228 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.594150    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.594228 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.594214    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddbn\" (UniqueName: \"kubernetes.io/projected/264c2b17-f41b-4127-9c1d-84c956d9706f-kube-api-access-nddbn\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.594357 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.594266    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:55:34.594357 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.594328    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-ca\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.596175 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.596146    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e860867b-f1e8-4d20-a11b-bb01c144fdc2-metrics-tls\") pod \"dns-default-lgnjx\" (UID: \"e860867b-f1e8-4d20-a11b-bb01c144fdc2\") " pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:55:34.596488 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.596469    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"image-registry-5b899cf4d4-r5nv9\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") " pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:55:34.611541 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.611521    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cj7w5\""
Sep 11 17:55:34.621635 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.621617    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:55:34.694784 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.694759    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddbn\" (UniqueName: \"kubernetes.io/projected/264c2b17-f41b-4127-9c1d-84c956d9706f-kube-api-access-nddbn\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.694906 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.694809    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-ca\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.694906 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.694831    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/264c2b17-f41b-4127-9c1d-84c956d9706f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.694906 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.694848    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-hub\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.694906 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.694865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.694906 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.694887    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.695822 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.695804    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/264c2b17-f41b-4127-9c1d-84c956d9706f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.697182 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.697151    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.697559 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.697542    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-ca\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.697878 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.697858    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.698068 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.698051    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/264c2b17-f41b-4127-9c1d-84c956d9706f-hub\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.710414 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.710394    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddbn\" (UniqueName: \"kubernetes.io/projected/264c2b17-f41b-4127-9c1d-84c956d9706f-kube-api-access-nddbn\") pod \"cluster-proxy-proxy-agent-6cccc4df9c-kz7mg\" (UID: \"264c2b17-f41b-4127-9c1d-84c956d9706f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.809486 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.809404    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"
Sep 11 17:55:34.886801 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.886777    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5487c\""
Sep 11 17:55:34.894991 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.894968    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:55:34.898049 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:34.898025    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lgnjx"]
Sep 11 17:55:34.901007 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:34.900985    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode860867b_f1e8_4d20_a11b_bb01c144fdc2.slice/crio-2841c9c3bb9f895e32c6a74a9252ec0f94e56008e8413bf5eb3372c1f627daba WatchSource:0}: Error finding container 2841c9c3bb9f895e32c6a74a9252ec0f94e56008e8413bf5eb3372c1f627daba: Status 404 returned error can't find the container with id 2841c9c3bb9f895e32c6a74a9252ec0f94e56008e8413bf5eb3372c1f627daba
Sep 11 17:55:35.085047 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:35.085023    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg"]
Sep 11 17:55:35.087361 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:35.087337    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod264c2b17_f41b_4127_9c1d_84c956d9706f.slice/crio-ec7f6f5bafa73b51f857dab66b86e08bac727d66aa44993a018dd93e567286dd WatchSource:0}: Error finding container ec7f6f5bafa73b51f857dab66b86e08bac727d66aa44993a018dd93e567286dd: Status 404 returned error can't find the container with id ec7f6f5bafa73b51f857dab66b86e08bac727d66aa44993a018dd93e567286dd
Sep 11 17:55:35.111565 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:35.111536    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lgnjx" event={"ID":"e860867b-f1e8-4d20-a11b-bb01c144fdc2","Type":"ContainerStarted","Data":"2841c9c3bb9f895e32c6a74a9252ec0f94e56008e8413bf5eb3372c1f627daba"}
Sep 11 17:55:35.112596 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:35.112578    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg" event={"ID":"264c2b17-f41b-4127-9c1d-84c956d9706f","Type":"ContainerStarted","Data":"ec7f6f5bafa73b51f857dab66b86e08bac727d66aa44993a018dd93e567286dd"}
Sep 11 17:55:35.182809 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:35.182778    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b899cf4d4-r5nv9"]
Sep 11 17:55:35.185830 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:35.185807    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdc3961_c5bf_48f8_b914_ca7a2981a9d7.slice/crio-f48860809195d29ca138ec2b6cc1fbea103e9b5d33fbfed9f7ecc17ddbf2bec2 WatchSource:0}: Error finding container f48860809195d29ca138ec2b6cc1fbea103e9b5d33fbfed9f7ecc17ddbf2bec2: Status 404 returned error can't find the container with id f48860809195d29ca138ec2b6cc1fbea103e9b5d33fbfed9f7ecc17ddbf2bec2
Sep 11 17:55:36.121279 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:36.121249    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9" event={"ID":"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7","Type":"ContainerStarted","Data":"b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5"}
Sep 11 17:55:36.121760 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:36.121322    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9" event={"ID":"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7","Type":"ContainerStarted","Data":"f48860809195d29ca138ec2b6cc1fbea103e9b5d33fbfed9f7ecc17ddbf2bec2"}
Sep 11 17:55:36.121760 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:36.121369    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:55:36.157411 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:36.157350    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9" podStartSLOduration=98.15733136 podStartE2EDuration="1m38.15733136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:36.151017846 +0000 UTC m=+99.921084208" watchObservedRunningTime="2026-09-11 17:55:36.15733136 +0000 UTC m=+99.927397703"
Sep 11 17:55:38.130911 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:38.130873    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lgnjx" event={"ID":"e860867b-f1e8-4d20-a11b-bb01c144fdc2","Type":"ContainerStarted","Data":"955eb32b9f96c001030e5fa03b63b9dc7a91dd7c6b3e1e8e4dc0552a7e49473b"}
Sep 11 17:55:38.131353 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:38.130974    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lgnjx" event={"ID":"e860867b-f1e8-4d20-a11b-bb01c144fdc2","Type":"ContainerStarted","Data":"f74b4ecc767b3e08a191a3488988473ba4043c351683d27ea860b5800026abf1"}
Sep 11 17:55:38.131353 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:38.130997    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:55:38.149560 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:38.149503    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lgnjx" podStartSLOduration=65.58736461 podStartE2EDuration="1m8.149485619s" totalImagesPullingTime="2.562121009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:38.148791125 +0000 UTC m=+101.918857476" watchObservedRunningTime="2026-09-11 17:55:38.149485619 +0000 UTC m=+101.919551967"
Sep 11 17:55:39.136644 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:39.136560    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg" event={"ID":"264c2b17-f41b-4127-9c1d-84c956d9706f","Type":"ContainerStarted","Data":"5af6a42effa7294997eeafd16df8ea97ce0c1e26f0f0c2d9540965ad2f86d6bd"}
Sep 11 17:55:39.136644 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:39.136602    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg" event={"ID":"264c2b17-f41b-4127-9c1d-84c956d9706f","Type":"ContainerStarted","Data":"ee694cb432bf6f863ea2cfb700b3319a1629f89ace0466eb29742f1331ecc330"}
Sep 11 17:55:39.136644 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:39.136616    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg" event={"ID":"264c2b17-f41b-4127-9c1d-84c956d9706f","Type":"ContainerStarted","Data":"070584a8f4bcdf3e25e55499e985b957f73a0415711bd906614059d3f4dd06f1"}
Sep 11 17:55:39.157617 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:39.157572    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6cccc4df9c-kz7mg" podStartSLOduration=1.474591603 podStartE2EDuration="5.157557406s" totalImagesPullingTime="3.682965803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:39.156773556 +0000 UTC m=+102.926839906" watchObservedRunningTime="2026-09-11 17:55:39.157557406 +0000 UTC m=+102.927623765"
Sep 11 17:55:41.019982 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:41.019955    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gkxjh"
Sep 11 17:55:43.331592 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.331555    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-snwm7"]
Sep 11 17:55:43.354730 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.354704    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.359576 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.359552    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 17:55:43.359576 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.359551    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 17:55:43.360607 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.360589    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:55:43.360750 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.360623    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:55:43.361228 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.361196    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 17:55:43.361228 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.361222    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:55:43.361417 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.361342    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-97vtp\""
Sep 11 17:55:43.462349 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462291    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-sys\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.462349 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462353    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-wtmp\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.462548 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-root\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.462548 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462425    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-textfile\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.462548 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462448    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-accelerators-collector-config\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.462548 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462466    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48aaac7f-0893-416d-99c3-f50d9aeb3ead-metrics-client-ca\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.462548 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462513    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-tls\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.462747 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462546    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7z4\" (UniqueName: \"kubernetes.io/projected/48aaac7f-0893-416d-99c3-f50d9aeb3ead-kube-api-access-lx7z4\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.462747 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.462584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563393 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563361    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-sys\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563426    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-wtmp\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563452    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-root\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-textfile\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563489    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-sys\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-accelerators-collector-config\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563523    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-root\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563574 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563540    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48aaac7f-0893-416d-99c3-f50d9aeb3ead-metrics-client-ca\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563950 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563590    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-wtmp\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563950 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-tls\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563950 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7z4\" (UniqueName: \"kubernetes.io/projected/48aaac7f-0893-416d-99c3-f50d9aeb3ead-kube-api-access-lx7z4\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.563950 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.563855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-textfile\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.564203 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.564180    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48aaac7f-0893-416d-99c3-f50d9aeb3ead-metrics-client-ca\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.564270 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.564180    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-accelerators-collector-config\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.565898 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.565874    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-tls\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.566007 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.565902    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48aaac7f-0893-416d-99c3-f50d9aeb3ead-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.573613 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.573591    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7z4\" (UniqueName: \"kubernetes.io/projected/48aaac7f-0893-416d-99c3-f50d9aeb3ead-kube-api-access-lx7z4\") pod \"node-exporter-snwm7\" (UID: \"48aaac7f-0893-416d-99c3-f50d9aeb3ead\") " pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.663576 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:43.663548    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-snwm7"
Sep 11 17:55:43.697954 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:55:43.697921    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48aaac7f_0893_416d_99c3_f50d9aeb3ead.slice/crio-9e20c656e26a82870bcbf44a701717a90548b559015df9b09ed6508ed03956ac WatchSource:0}: Error finding container 9e20c656e26a82870bcbf44a701717a90548b559015df9b09ed6508ed03956ac: Status 404 returned error can't find the container with id 9e20c656e26a82870bcbf44a701717a90548b559015df9b09ed6508ed03956ac
Sep 11 17:55:44.155661 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:44.155626    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-snwm7" event={"ID":"48aaac7f-0893-416d-99c3-f50d9aeb3ead","Type":"ContainerStarted","Data":"9e20c656e26a82870bcbf44a701717a90548b559015df9b09ed6508ed03956ac"}
Sep 11 17:55:45.160517 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:45.160487    2571 generic.go:356] "Generic (PLEG): container finished" podID="48aaac7f-0893-416d-99c3-f50d9aeb3ead" containerID="5d44b0373dc338c8e67c008eae95d0cf26a2be70dcc5a5fe6509020f5b752468" exitCode=0
Sep 11 17:55:45.160884 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:45.160567    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-snwm7" event={"ID":"48aaac7f-0893-416d-99c3-f50d9aeb3ead","Type":"ContainerDied","Data":"5d44b0373dc338c8e67c008eae95d0cf26a2be70dcc5a5fe6509020f5b752468"}
Sep 11 17:55:46.166336 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:46.166283    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-snwm7" event={"ID":"48aaac7f-0893-416d-99c3-f50d9aeb3ead","Type":"ContainerStarted","Data":"73cc1df002f7267a78bb886faa6b8730163857efa5367b799f6e0d3d4b31cc90"}
Sep 11 17:55:46.166740 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:46.166383    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-snwm7" event={"ID":"48aaac7f-0893-416d-99c3-f50d9aeb3ead","Type":"ContainerStarted","Data":"293dfb17b84eb4abff70437eb8003270f0527292eb66d0861ec6f11e42a22553"}
Sep 11 17:55:46.187796 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:46.187749    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-snwm7" podStartSLOduration=1.854655401 podStartE2EDuration="3.187736513s" totalImagesPullingTime="1.262065712s" totalInitContainerRuntime="71.0154ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:46.186274601 +0000 UTC m=+109.956340949" watchObservedRunningTime="2026-09-11 17:55:46.187736513 +0000 UTC m=+109.957802860"
Sep 11 17:55:48.139499 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:48.139461    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lgnjx"
Sep 11 17:55:56.989240 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:56.989194    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b899cf4d4-r5nv9"]
Sep 11 17:55:56.993416 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:55:56.993394    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:56:06.534582 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:06.534525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:56:06.536845 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:06.536824    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/139a5b42-f2cc-4f37-9cac-8e7ab4ab4373-metrics-certs\") pod \"network-metrics-daemon-hl9ct\" (UID: \"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373\") " pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:56:06.554815 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:06.554788    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-64859\""
Sep 11 17:56:06.561975 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:06.561955    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hl9ct"
Sep 11 17:56:06.852251 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:06.852145    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hl9ct"]
Sep 11 17:56:06.854331 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:56:06.854288    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139a5b42_f2cc_4f37_9cac_8e7ab4ab4373.slice/crio-c87e6d0acf8b7c72f00661fab6a46d23cf7c98028c30dad102227db4d6ea37f8 WatchSource:0}: Error finding container c87e6d0acf8b7c72f00661fab6a46d23cf7c98028c30dad102227db4d6ea37f8: Status 404 returned error can't find the container with id c87e6d0acf8b7c72f00661fab6a46d23cf7c98028c30dad102227db4d6ea37f8
Sep 11 17:56:07.232252 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:07.232219    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hl9ct" event={"ID":"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373","Type":"ContainerStarted","Data":"c87e6d0acf8b7c72f00661fab6a46d23cf7c98028c30dad102227db4d6ea37f8"}
Sep 11 17:56:08.236832 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:08.236800    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hl9ct" event={"ID":"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373","Type":"ContainerStarted","Data":"42fbe66b213b98bfd7dcfbf9d69489800eeba131b393d53f0edc525c34bcb56f"}
Sep 11 17:56:08.237161 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:08.236852    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hl9ct" event={"ID":"139a5b42-f2cc-4f37-9cac-8e7ab4ab4373","Type":"ContainerStarted","Data":"a9179145b38bd491d8c0b76c485b1bdd12f18042e4ae7c1acc89c49733e0eadd"}
Sep 11 17:56:08.255091 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:08.255050    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hl9ct" podStartSLOduration=130.098396098 podStartE2EDuration="2m11.255035864s" totalImagesPullingTime="1.156639766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:08.253843751 +0000 UTC m=+132.023910111" watchObservedRunningTime="2026-09-11 17:56:08.255035864 +0000 UTC m=+132.025102211"
Sep 11 17:56:18.121546 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:18.121519    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hn9nt_065a0aab-56f6-461e-bfb1-e9a51a830677/serve-healthcheck-canary/0.log"
Sep 11 17:56:20.279063 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:20.279029    2571 generic.go:356] "Generic (PLEG): container finished" podID="14a35285-829b-490a-a879-8b415812e26a" containerID="5cec127a1af01b44fbf9da5a32e2b0c100515914881d036534740c0ff75f5e7e" exitCode=0
Sep 11 17:56:20.279551 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:20.279083    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls" event={"ID":"14a35285-829b-490a-a879-8b415812e26a","Type":"ContainerDied","Data":"5cec127a1af01b44fbf9da5a32e2b0c100515914881d036534740c0ff75f5e7e"}
Sep 11 17:56:20.279551 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:20.279458    2571 scope.go:117] "RemoveContainer" containerID="5cec127a1af01b44fbf9da5a32e2b0c100515914881d036534740c0ff75f5e7e"
Sep 11 17:56:21.284177 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:21.284138    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-bzsls" event={"ID":"14a35285-829b-490a-a879-8b415812e26a","Type":"ContainerStarted","Data":"179bd1d0928f2dc91b16bd7644e52b97891f60608192bd6793915b2f2de14481"}
Sep 11 17:56:22.007673 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.007616    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9" podUID="cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" containerName="registry" containerID="cri-o://b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5" gracePeriod=30
Sep 11 17:56:22.245785 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.245763    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:56:22.289624 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.289547    2571 generic.go:356] "Generic (PLEG): container finished" podID="cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" containerID="b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5" exitCode=0
Sep 11 17:56:22.289624 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.289584    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9"
Sep 11 17:56:22.289988 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.289629    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9" event={"ID":"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7","Type":"ContainerDied","Data":"b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5"}
Sep 11 17:56:22.289988 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.289676    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b899cf4d4-r5nv9" event={"ID":"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7","Type":"ContainerDied","Data":"f48860809195d29ca138ec2b6cc1fbea103e9b5d33fbfed9f7ecc17ddbf2bec2"}
Sep 11 17:56:22.289988 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.289701    2571 scope.go:117] "RemoveContainer" containerID="b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5"
Sep 11 17:56:22.297280 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.297262    2571 scope.go:117] "RemoveContainer" containerID="b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5"
Sep 11 17:56:22.297579 ip-10-0-133-109 kubenswrapper[2571]: E0911 17:56:22.297555    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5\": container with ID starting with b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5 not found: ID does not exist" containerID="b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5"
Sep 11 17:56:22.297682 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.297586    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5"} err="failed to get container status \"b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5\": rpc error: code = NotFound desc = could not find container \"b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5\": container with ID starting with b529dfb33469a6593c64f9e19bab15f2569597c72fd2401f2577011f1d1009e5 not found: ID does not exist"
Sep 11 17:56:22.329864 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.329842    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-image-registry-private-configuration\") pod \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") "
Sep 11 17:56:22.330021 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.329880    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-trusted-ca\") pod \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") "
Sep 11 17:56:22.330021 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.329903    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-ca-trust-extracted\") pod \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") "
Sep 11 17:56:22.330021 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.329928    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-certificates\") pod \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") "
Sep 11 17:56:22.330021 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.329955    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-installation-pull-secrets\") pod \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") "
Sep 11 17:56:22.330021 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.330012    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-kube-api-access-td49m\") pod \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") "
Sep 11 17:56:22.330272 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.330049    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") pod \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") "
Sep 11 17:56:22.330272 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.330082    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-bound-sa-token\") pod \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\" (UID: \"cbdc3961-c5bf-48f8-b914-ca7a2981a9d7\") "
Sep 11 17:56:22.330469 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.330443    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:22.330697 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.330657    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:22.332510 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.332480    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:22.332510 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.332495    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:22.332658 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.332578    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:22.332743 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.332718    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-kube-api-access-td49m" (OuterVolumeSpecName: "kube-api-access-td49m") pod "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7"). InnerVolumeSpecName "kube-api-access-td49m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:22.332828 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.332800    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:22.338482 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.338460    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" (UID: "cbdc3961-c5bf-48f8-b914-ca7a2981a9d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:56:22.431371 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.431339    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-tls\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:56:22.431371 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.431365    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-bound-sa-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:56:22.431371 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.431374    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-image-registry-private-configuration\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:56:22.431584 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.431384    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-trusted-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:56:22.431584 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.431392    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-ca-trust-extracted\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:56:22.431584 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.431411    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-registry-certificates\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:56:22.431584 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.431421    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-installation-pull-secrets\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:56:22.431584 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.431429    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td49m\" (UniqueName: \"kubernetes.io/projected/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7-kube-api-access-td49m\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:56:22.611031 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.610999    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b899cf4d4-r5nv9"]
Sep 11 17:56:22.613459 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.613437    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b899cf4d4-r5nv9"]
Sep 11 17:56:22.728915 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:56:22.728886    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" path="/var/lib/kubelet/pods/cbdc3961-c5bf-48f8-b914-ca7a2981a9d7/volumes"
Sep 11 17:58:56.662664 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:58:56.662636    2571 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 17:59:02.469488 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.469457    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"]
Sep 11 17:59:02.471709 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.469691    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" containerName="registry"
Sep 11 17:59:02.471709 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.469702    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" containerName="registry"
Sep 11 17:59:02.471709 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.469755    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbdc3961-c5bf-48f8-b914-ca7a2981a9d7" containerName="registry"
Sep 11 17:59:02.472616 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.472598    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.479023 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.479005    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:59:02.480832 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.480816    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p767k\""
Sep 11 17:59:02.481101 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.481085    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:59:02.512144 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.512115    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"]
Sep 11 17:59:02.605243 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.605201    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc468\" (UniqueName: \"kubernetes.io/projected/f40d7988-58ed-43dc-872b-6a38f4b4e177-kube-api-access-gc468\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.605427 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.605274    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.605427 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.605324    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.705620 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.705578    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc468\" (UniqueName: \"kubernetes.io/projected/f40d7988-58ed-43dc-872b-6a38f4b4e177-kube-api-access-gc468\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.705755 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.705639    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.705755 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.705671    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.706153 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.706133    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.706153 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.706145    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.716163 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.716134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc468\" (UniqueName: \"kubernetes.io/projected/f40d7988-58ed-43dc-872b-6a38f4b4e177-kube-api-access-gc468\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:02.780991 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:02.780907    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:03.058379 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:03.058291    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"]
Sep 11 17:59:03.062802 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:59:03.062772    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40d7988_58ed_43dc_872b_6a38f4b4e177.slice/crio-b1c7a34a7366ac5615484501f96e4857934f55b6679513d081e80cbfaf7d6637 WatchSource:0}: Error finding container b1c7a34a7366ac5615484501f96e4857934f55b6679513d081e80cbfaf7d6637: Status 404 returned error can't find the container with id b1c7a34a7366ac5615484501f96e4857934f55b6679513d081e80cbfaf7d6637
Sep 11 17:59:03.064815 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:03.064797    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:59:03.811127 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:03.811097    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db" event={"ID":"f40d7988-58ed-43dc-872b-6a38f4b4e177","Type":"ContainerStarted","Data":"b1c7a34a7366ac5615484501f96e4857934f55b6679513d081e80cbfaf7d6637"}
Sep 11 17:59:09.832641 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:09.832605    2571 generic.go:356] "Generic (PLEG): container finished" podID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerID="6f3fe0a2016a3689f4d9a9f6fad8de71a15589e7f39a7856580d8a7ef0a3998d" exitCode=0
Sep 11 17:59:09.833056 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:09.832659    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db" event={"ID":"f40d7988-58ed-43dc-872b-6a38f4b4e177","Type":"ContainerDied","Data":"6f3fe0a2016a3689f4d9a9f6fad8de71a15589e7f39a7856580d8a7ef0a3998d"}
Sep 11 17:59:12.843998 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:12.843959    2571 generic.go:356] "Generic (PLEG): container finished" podID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerID="84612b13917772d543b032117a9fc5b7d391a209d869552425be8c8aca4e0076" exitCode=0
Sep 11 17:59:12.843998 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:12.844001    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db" event={"ID":"f40d7988-58ed-43dc-872b-6a38f4b4e177","Type":"ContainerDied","Data":"84612b13917772d543b032117a9fc5b7d391a209d869552425be8c8aca4e0076"}
Sep 11 17:59:19.871810 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:19.871731    2571 generic.go:356] "Generic (PLEG): container finished" podID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerID="6e51734f4fdf320fad111ff95779645b74b3a40f708295440d871ee34df376dc" exitCode=0
Sep 11 17:59:19.871810 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:19.871774    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db" event={"ID":"f40d7988-58ed-43dc-872b-6a38f4b4e177","Type":"ContainerDied","Data":"6e51734f4fdf320fad111ff95779645b74b3a40f708295440d871ee34df376dc"}
Sep 11 17:59:20.996411 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:20.996388    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:21.143186 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.143095    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-bundle\") pod \"f40d7988-58ed-43dc-872b-6a38f4b4e177\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") "
Sep 11 17:59:21.143186 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.143161    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-util\") pod \"f40d7988-58ed-43dc-872b-6a38f4b4e177\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") "
Sep 11 17:59:21.143424 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.143204    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc468\" (UniqueName: \"kubernetes.io/projected/f40d7988-58ed-43dc-872b-6a38f4b4e177-kube-api-access-gc468\") pod \"f40d7988-58ed-43dc-872b-6a38f4b4e177\" (UID: \"f40d7988-58ed-43dc-872b-6a38f4b4e177\") "
Sep 11 17:59:21.143782 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.143755    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-bundle" (OuterVolumeSpecName: "bundle") pod "f40d7988-58ed-43dc-872b-6a38f4b4e177" (UID: "f40d7988-58ed-43dc-872b-6a38f4b4e177"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:59:21.145504 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.145474    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40d7988-58ed-43dc-872b-6a38f4b4e177-kube-api-access-gc468" (OuterVolumeSpecName: "kube-api-access-gc468") pod "f40d7988-58ed-43dc-872b-6a38f4b4e177" (UID: "f40d7988-58ed-43dc-872b-6a38f4b4e177"). InnerVolumeSpecName "kube-api-access-gc468". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:59:21.148261 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.148240    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-util" (OuterVolumeSpecName: "util") pod "f40d7988-58ed-43dc-872b-6a38f4b4e177" (UID: "f40d7988-58ed-43dc-872b-6a38f4b4e177"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:59:21.243983 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.243924    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-util\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:59:21.243983 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.243974    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc468\" (UniqueName: \"kubernetes.io/projected/f40d7988-58ed-43dc-872b-6a38f4b4e177-kube-api-access-gc468\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:59:21.243983 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.243985    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f40d7988-58ed-43dc-872b-6a38f4b4e177-bundle\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:59:21.880063 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.880025    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db" event={"ID":"f40d7988-58ed-43dc-872b-6a38f4b4e177","Type":"ContainerDied","Data":"b1c7a34a7366ac5615484501f96e4857934f55b6679513d081e80cbfaf7d6637"}
Sep 11 17:59:21.880063 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.880050    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540th4db"
Sep 11 17:59:21.880266 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:21.880078    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1c7a34a7366ac5615484501f96e4857934f55b6679513d081e80cbfaf7d6637"
Sep 11 17:59:25.497015 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.496979    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"]
Sep 11 17:59:25.497578 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.497338    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerName="util"
Sep 11 17:59:25.497578 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.497355    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerName="util"
Sep 11 17:59:25.497578 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.497368    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerName="extract"
Sep 11 17:59:25.497578 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.497376    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerName="extract"
Sep 11 17:59:25.497578 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.497387    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerName="pull"
Sep 11 17:59:25.497578 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.497394    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerName="pull"
Sep 11 17:59:25.497578 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.497469    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40d7988-58ed-43dc-872b-6a38f4b4e177" containerName="extract"
Sep 11 17:59:25.538600 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.538563    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"]
Sep 11 17:59:25.538747 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.538679    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"
Sep 11 17:59:25.541898 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.541876    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-442nn\""
Sep 11 17:59:25.541996 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.541909    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:59:25.541996 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.541880    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 11 17:59:25.672154 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.672123    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cgh4\" (UniqueName: \"kubernetes.io/projected/9efbeb35-4180-4e84-b8bf-c39e3867cc8f-kube-api-access-7cgh4\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr5xq\" (UID: \"9efbeb35-4180-4e84-b8bf-c39e3867cc8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"
Sep 11 17:59:25.672358 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.672164    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efbeb35-4180-4e84-b8bf-c39e3867cc8f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr5xq\" (UID: \"9efbeb35-4180-4e84-b8bf-c39e3867cc8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"
Sep 11 17:59:25.773587 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.773492    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgh4\" (UniqueName: \"kubernetes.io/projected/9efbeb35-4180-4e84-b8bf-c39e3867cc8f-kube-api-access-7cgh4\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr5xq\" (UID: \"9efbeb35-4180-4e84-b8bf-c39e3867cc8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"
Sep 11 17:59:25.773587 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.773547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efbeb35-4180-4e84-b8bf-c39e3867cc8f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr5xq\" (UID: \"9efbeb35-4180-4e84-b8bf-c39e3867cc8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"
Sep 11 17:59:25.773920 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.773901    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9efbeb35-4180-4e84-b8bf-c39e3867cc8f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr5xq\" (UID: \"9efbeb35-4180-4e84-b8bf-c39e3867cc8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"
Sep 11 17:59:25.782376 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.782349    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgh4\" (UniqueName: \"kubernetes.io/projected/9efbeb35-4180-4e84-b8bf-c39e3867cc8f-kube-api-access-7cgh4\") pod \"cert-manager-operator-controller-manager-55c444bf86-rr5xq\" (UID: \"9efbeb35-4180-4e84-b8bf-c39e3867cc8f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"
Sep 11 17:59:25.847704 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:25.847666    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"
Sep 11 17:59:26.133870 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:26.133846    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq"]
Sep 11 17:59:26.135939 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:59:26.135914    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efbeb35_4180_4e84_b8bf_c39e3867cc8f.slice/crio-99b737bc4c390c08e63c1824c58a6d79e9d0aa9d5606067e466bff5496bfcb53 WatchSource:0}: Error finding container 99b737bc4c390c08e63c1824c58a6d79e9d0aa9d5606067e466bff5496bfcb53: Status 404 returned error can't find the container with id 99b737bc4c390c08e63c1824c58a6d79e9d0aa9d5606067e466bff5496bfcb53
Sep 11 17:59:26.900256 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:26.900220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq" event={"ID":"9efbeb35-4180-4e84-b8bf-c39e3867cc8f","Type":"ContainerStarted","Data":"99b737bc4c390c08e63c1824c58a6d79e9d0aa9d5606067e466bff5496bfcb53"}
Sep 11 17:59:28.908069 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:28.908036    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq" event={"ID":"9efbeb35-4180-4e84-b8bf-c39e3867cc8f","Type":"ContainerStarted","Data":"3d9937765156ad09d34f141753e1be42ef08bc49b7162f4cd6e42e4ec7464294"}
Sep 11 17:59:28.933658 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:28.933605    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-rr5xq" podStartSLOduration=1.730656323 podStartE2EDuration="3.933586344s" totalImagesPullingTime="2.202930021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:28.932086061 +0000 UTC m=+332.702152411" watchObservedRunningTime="2026-09-11 17:59:28.933586344 +0000 UTC m=+332.703652694"
Sep 11 17:59:29.571914 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.571879    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"]
Sep 11 17:59:29.594620 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.594587    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"]
Sep 11 17:59:29.594761 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.594700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.597518 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.597495    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:59:29.597642 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.597507    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p767k\""
Sep 11 17:59:29.598618 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.598602    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:59:29.700831 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.700798    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.700987 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.700866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.700987 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.700899    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnz8v\" (UniqueName: \"kubernetes.io/projected/116d2fc9-3c96-41ba-adf6-66915ab56fbe-kube-api-access-xnz8v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.801947 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.801914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.801947 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.801976    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.802184 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.801998    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnz8v\" (UniqueName: \"kubernetes.io/projected/116d2fc9-3c96-41ba-adf6-66915ab56fbe-kube-api-access-xnz8v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.802405 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.802382    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.802515 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.802492    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.811522 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.811501    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnz8v\" (UniqueName: \"kubernetes.io/projected/116d2fc9-3c96-41ba-adf6-66915ab56fbe-kube-api-access-xnz8v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:29.903420 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:29.903362    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:30.176000 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:30.175936    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"]
Sep 11 17:59:30.177911 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:59:30.177886    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116d2fc9_3c96_41ba_adf6_66915ab56fbe.slice/crio-7af236c927a3d733be54a542946c3370044b498cab3aab221ea192603bf5c838 WatchSource:0}: Error finding container 7af236c927a3d733be54a542946c3370044b498cab3aab221ea192603bf5c838: Status 404 returned error can't find the container with id 7af236c927a3d733be54a542946c3370044b498cab3aab221ea192603bf5c838
Sep 11 17:59:30.917847 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:30.917811    2571 generic.go:356] "Generic (PLEG): container finished" podID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerID="c0c48906b6922b5a1b9cb0f1e116b2ab44a60cb572b77b9794034fd988fc614a" exitCode=0
Sep 11 17:59:30.917998 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:30.917850    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh" event={"ID":"116d2fc9-3c96-41ba-adf6-66915ab56fbe","Type":"ContainerDied","Data":"c0c48906b6922b5a1b9cb0f1e116b2ab44a60cb572b77b9794034fd988fc614a"}
Sep 11 17:59:30.917998 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:30.917887    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh" event={"ID":"116d2fc9-3c96-41ba-adf6-66915ab56fbe","Type":"ContainerStarted","Data":"7af236c927a3d733be54a542946c3370044b498cab3aab221ea192603bf5c838"}
Sep 11 17:59:32.928387 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:32.928352    2571 generic.go:356] "Generic (PLEG): container finished" podID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerID="8015e525b5bcddf732034f5dfad6f11cf8a330dcc0732de500cf8666f36f0c94" exitCode=0
Sep 11 17:59:32.928749 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:32.928443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh" event={"ID":"116d2fc9-3c96-41ba-adf6-66915ab56fbe","Type":"ContainerDied","Data":"8015e525b5bcddf732034f5dfad6f11cf8a330dcc0732de500cf8666f36f0c94"}
Sep 11 17:59:33.933746 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:33.933710    2571 generic.go:356] "Generic (PLEG): container finished" podID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerID="1294c29ee47c966a4ea50ca30ef832a527b8b96a69a59a0eb7e2fe6cff0c76a0" exitCode=0
Sep 11 17:59:33.934130 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:33.933756    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh" event={"ID":"116d2fc9-3c96-41ba-adf6-66915ab56fbe","Type":"ContainerDied","Data":"1294c29ee47c966a4ea50ca30ef832a527b8b96a69a59a0eb7e2fe6cff0c76a0"}
Sep 11 17:59:34.659783 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.658089    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"]
Sep 11 17:59:34.661923 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.661894    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"
Sep 11 17:59:34.664772 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.664745    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 17:59:34.664772 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.664760    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 17:59:34.664772 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.664745    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vw7vm\""
Sep 11 17:59:34.667890 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.667869    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"]
Sep 11 17:59:34.838198 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.838159    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lknpn\" (UniqueName: \"kubernetes.io/projected/22552725-6aa6-4437-baea-c871c7453016-kube-api-access-lknpn\") pod \"cert-manager-cainjector-559cdd8c87-25jnn\" (UID: \"22552725-6aa6-4437-baea-c871c7453016\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"
Sep 11 17:59:34.838393 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.838217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22552725-6aa6-4437-baea-c871c7453016-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-25jnn\" (UID: \"22552725-6aa6-4437-baea-c871c7453016\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"
Sep 11 17:59:34.938849 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.938761    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lknpn\" (UniqueName: \"kubernetes.io/projected/22552725-6aa6-4437-baea-c871c7453016-kube-api-access-lknpn\") pod \"cert-manager-cainjector-559cdd8c87-25jnn\" (UID: \"22552725-6aa6-4437-baea-c871c7453016\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"
Sep 11 17:59:34.938849 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.938835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22552725-6aa6-4437-baea-c871c7453016-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-25jnn\" (UID: \"22552725-6aa6-4437-baea-c871c7453016\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"
Sep 11 17:59:34.950554 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.950522    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/22552725-6aa6-4437-baea-c871c7453016-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-25jnn\" (UID: \"22552725-6aa6-4437-baea-c871c7453016\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"
Sep 11 17:59:34.950691 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.950612    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lknpn\" (UniqueName: \"kubernetes.io/projected/22552725-6aa6-4437-baea-c871c7453016-kube-api-access-lknpn\") pod \"cert-manager-cainjector-559cdd8c87-25jnn\" (UID: \"22552725-6aa6-4437-baea-c871c7453016\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"
Sep 11 17:59:34.972208 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:34.972178    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"
Sep 11 17:59:35.065171 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.065149    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:35.242096 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.242031    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnz8v\" (UniqueName: \"kubernetes.io/projected/116d2fc9-3c96-41ba-adf6-66915ab56fbe-kube-api-access-xnz8v\") pod \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") "
Sep 11 17:59:35.242096 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.242093    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-bundle\") pod \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") "
Sep 11 17:59:35.242279 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.242135    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-util\") pod \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\" (UID: \"116d2fc9-3c96-41ba-adf6-66915ab56fbe\") "
Sep 11 17:59:35.242482 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.242456    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-bundle" (OuterVolumeSpecName: "bundle") pod "116d2fc9-3c96-41ba-adf6-66915ab56fbe" (UID: "116d2fc9-3c96-41ba-adf6-66915ab56fbe"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:59:35.244011 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.243986    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116d2fc9-3c96-41ba-adf6-66915ab56fbe-kube-api-access-xnz8v" (OuterVolumeSpecName: "kube-api-access-xnz8v") pod "116d2fc9-3c96-41ba-adf6-66915ab56fbe" (UID: "116d2fc9-3c96-41ba-adf6-66915ab56fbe"). InnerVolumeSpecName "kube-api-access-xnz8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:59:35.247699 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.247672    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-util" (OuterVolumeSpecName: "util") pod "116d2fc9-3c96-41ba-adf6-66915ab56fbe" (UID: "116d2fc9-3c96-41ba-adf6-66915ab56fbe"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:59:35.286142 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.286118    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-25jnn"]
Sep 11 17:59:35.287780 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:59:35.287743    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22552725_6aa6_4437_baea_c871c7453016.slice/crio-d97b711cc52989e99c32a7a01e292174ae3aba891f8346ffe38722dab682c82a WatchSource:0}: Error finding container d97b711cc52989e99c32a7a01e292174ae3aba891f8346ffe38722dab682c82a: Status 404 returned error can't find the container with id d97b711cc52989e99c32a7a01e292174ae3aba891f8346ffe38722dab682c82a
Sep 11 17:59:35.342906 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.342873    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-bundle\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:59:35.342906 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.342900    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/116d2fc9-3c96-41ba-adf6-66915ab56fbe-util\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:59:35.343090 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.342913    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnz8v\" (UniqueName: \"kubernetes.io/projected/116d2fc9-3c96-41ba-adf6-66915ab56fbe-kube-api-access-xnz8v\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 17:59:35.945590 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.945560    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh"
Sep 11 17:59:35.946155 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.945562    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f8rsvh" event={"ID":"116d2fc9-3c96-41ba-adf6-66915ab56fbe","Type":"ContainerDied","Data":"7af236c927a3d733be54a542946c3370044b498cab3aab221ea192603bf5c838"}
Sep 11 17:59:35.946155 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.945689    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7af236c927a3d733be54a542946c3370044b498cab3aab221ea192603bf5c838"
Sep 11 17:59:35.947054 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:35.947030    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn" event={"ID":"22552725-6aa6-4437-baea-c871c7453016","Type":"ContainerStarted","Data":"d97b711cc52989e99c32a7a01e292174ae3aba891f8346ffe38722dab682c82a"}
Sep 11 17:59:37.956414 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:37.956374    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn" event={"ID":"22552725-6aa6-4437-baea-c871c7453016","Type":"ContainerStarted","Data":"5181a5f30d501e6f9a7d48519c31f58d8760bf7420b7aeed33da62f6ca870a20"}
Sep 11 17:59:37.988277 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:37.988223    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-25jnn" podStartSLOduration=1.50491606 podStartE2EDuration="3.988206765s" totalImagesPullingTime="2.483290705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:37.986848099 +0000 UTC m=+341.756914440" watchObservedRunningTime="2026-09-11 17:59:37.988206765 +0000 UTC m=+341.758273124"
Sep 11 17:59:42.494901 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.494864    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"]
Sep 11 17:59:42.495315 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.495117    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerName="pull"
Sep 11 17:59:42.495315 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.495127    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerName="pull"
Sep 11 17:59:42.495315 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.495140    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerName="util"
Sep 11 17:59:42.495315 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.495145    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerName="util"
Sep 11 17:59:42.495315 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.495152    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerName="extract"
Sep 11 17:59:42.495315 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.495157    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerName="extract"
Sep 11 17:59:42.495315 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.495209    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="116d2fc9-3c96-41ba-adf6-66915ab56fbe" containerName="extract"
Sep 11 17:59:42.498041 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.498026    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"
Sep 11 17:59:42.500861 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.500844    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:59:42.502028 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.502012    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 17:59:42.502146 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.502130    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2p8hk\""
Sep 11 17:59:42.506987 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.506963    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"]
Sep 11 17:59:42.591851 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.591816    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqv9s\" (UniqueName: \"kubernetes.io/projected/4a21436e-4ff5-4b36-b48d-fb77f3b84f04-kube-api-access-vqv9s\") pod \"openshift-lws-operator-fd8ccff4c-79h58\" (UID: \"4a21436e-4ff5-4b36-b48d-fb77f3b84f04\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"
Sep 11 17:59:42.591851 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.591858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a21436e-4ff5-4b36-b48d-fb77f3b84f04-tmp\") pod \"openshift-lws-operator-fd8ccff4c-79h58\" (UID: \"4a21436e-4ff5-4b36-b48d-fb77f3b84f04\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"
Sep 11 17:59:42.692511 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.692476    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqv9s\" (UniqueName: \"kubernetes.io/projected/4a21436e-4ff5-4b36-b48d-fb77f3b84f04-kube-api-access-vqv9s\") pod \"openshift-lws-operator-fd8ccff4c-79h58\" (UID: \"4a21436e-4ff5-4b36-b48d-fb77f3b84f04\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"
Sep 11 17:59:42.692511 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.692522    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a21436e-4ff5-4b36-b48d-fb77f3b84f04-tmp\") pod \"openshift-lws-operator-fd8ccff4c-79h58\" (UID: \"4a21436e-4ff5-4b36-b48d-fb77f3b84f04\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"
Sep 11 17:59:42.692898 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.692877    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a21436e-4ff5-4b36-b48d-fb77f3b84f04-tmp\") pod \"openshift-lws-operator-fd8ccff4c-79h58\" (UID: \"4a21436e-4ff5-4b36-b48d-fb77f3b84f04\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"
Sep 11 17:59:42.701554 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.701527    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqv9s\" (UniqueName: \"kubernetes.io/projected/4a21436e-4ff5-4b36-b48d-fb77f3b84f04-kube-api-access-vqv9s\") pod \"openshift-lws-operator-fd8ccff4c-79h58\" (UID: \"4a21436e-4ff5-4b36-b48d-fb77f3b84f04\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"
Sep 11 17:59:42.806688 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:42.806596    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"
Sep 11 17:59:43.079614 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:43.079591    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58"]
Sep 11 17:59:43.082399 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:59:43.082371    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a21436e_4ff5_4b36_b48d_fb77f3b84f04.slice/crio-114e77fca75312ea68e8234f26921d292ccfce92d282f79325333a1918dc4f13 WatchSource:0}: Error finding container 114e77fca75312ea68e8234f26921d292ccfce92d282f79325333a1918dc4f13: Status 404 returned error can't find the container with id 114e77fca75312ea68e8234f26921d292ccfce92d282f79325333a1918dc4f13
Sep 11 17:59:43.980699 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:43.980663    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58" event={"ID":"4a21436e-4ff5-4b36-b48d-fb77f3b84f04","Type":"ContainerStarted","Data":"114e77fca75312ea68e8234f26921d292ccfce92d282f79325333a1918dc4f13"}
Sep 11 17:59:57.036444 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:57.036402    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58" event={"ID":"4a21436e-4ff5-4b36-b48d-fb77f3b84f04","Type":"ContainerStarted","Data":"f7434ec0f261a08dffd2bdc04aba170cf94f463e3c7cacfbbf71676961b3bae8"}
Sep 11 17:59:57.056263 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:57.056211    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79h58" podStartSLOduration=1.901721764 podStartE2EDuration="15.056195311s" totalImagesPullingTime="13.154473547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:57.054602374 +0000 UTC m=+360.824668725" watchObservedRunningTime="2026-09-11 17:59:57.056195311 +0000 UTC m=+360.826261662"
Sep 11 17:59:59.323649 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.323605    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"]
Sep 11 17:59:59.327155 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.327132    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.330468 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.330444    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p767k\""
Sep 11 17:59:59.330556 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.330444    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 17:59:59.331744 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.331728    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 17:59:59.341752 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.341728    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"]
Sep 11 17:59:59.401519 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.401487    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.401519 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.401533    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blchm\" (UniqueName: \"kubernetes.io/projected/a34e6738-b233-4ba6-aeb7-f301cedca59a-kube-api-access-blchm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.401732 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.401578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.502531 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.502501    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.502686 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.502545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.502686 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.502577    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blchm\" (UniqueName: \"kubernetes.io/projected/a34e6738-b233-4ba6-aeb7-f301cedca59a-kube-api-access-blchm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.502832 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.502813    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.502932 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.502910    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.512614 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.512588    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blchm\" (UniqueName: \"kubernetes.io/projected/a34e6738-b233-4ba6-aeb7-f301cedca59a-kube-api-access-blchm\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.636257 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.636168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 17:59:59.909639 ip-10-0-133-109 kubenswrapper[2571]: I0911 17:59:59.909610    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"]
Sep 11 17:59:59.911761 ip-10-0-133-109 kubenswrapper[2571]: W0911 17:59:59.911740    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34e6738_b233_4ba6_aeb7_f301cedca59a.slice/crio-d01c09c7e893deff68ed8f63834c37aeb19433c767bd8355bc35ca0f4531dd30 WatchSource:0}: Error finding container d01c09c7e893deff68ed8f63834c37aeb19433c767bd8355bc35ca0f4531dd30: Status 404 returned error can't find the container with id d01c09c7e893deff68ed8f63834c37aeb19433c767bd8355bc35ca0f4531dd30
Sep 11 18:00:00.049165 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:00.049133    2571 generic.go:356] "Generic (PLEG): container finished" podID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerID="2e14576ef5bba50a44509d74ce1f54467e27e697efde446e7fc59e00a26d56d6" exitCode=0
Sep 11 18:00:00.049353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:00.049168    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd" event={"ID":"a34e6738-b233-4ba6-aeb7-f301cedca59a","Type":"ContainerDied","Data":"2e14576ef5bba50a44509d74ce1f54467e27e697efde446e7fc59e00a26d56d6"}
Sep 11 18:00:00.049353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:00.049217    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd" event={"ID":"a34e6738-b233-4ba6-aeb7-f301cedca59a","Type":"ContainerStarted","Data":"d01c09c7e893deff68ed8f63834c37aeb19433c767bd8355bc35ca0f4531dd30"}
Sep 11 18:00:01.055048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:01.054959    2571 generic.go:356] "Generic (PLEG): container finished" podID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerID="ab95a842f02eed984259842bf3888a28d325455962b456299f017cc6582cc1ff" exitCode=0
Sep 11 18:00:01.055048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:01.055002    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd" event={"ID":"a34e6738-b233-4ba6-aeb7-f301cedca59a","Type":"ContainerDied","Data":"ab95a842f02eed984259842bf3888a28d325455962b456299f017cc6582cc1ff"}
Sep 11 18:00:02.061077 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:02.061039    2571 generic.go:356] "Generic (PLEG): container finished" podID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerID="9d59e613c9c5d55eb57accecb08affafd9417aa0dafd9e91891f7105f1436216" exitCode=0
Sep 11 18:00:02.061486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:02.061088    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd" event={"ID":"a34e6738-b233-4ba6-aeb7-f301cedca59a","Type":"ContainerDied","Data":"9d59e613c9c5d55eb57accecb08affafd9417aa0dafd9e91891f7105f1436216"}
Sep 11 18:00:03.186498 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.186470    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 18:00:03.228874 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.228840    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-bundle\") pod \"a34e6738-b233-4ba6-aeb7-f301cedca59a\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") "
Sep 11 18:00:03.229041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.228896    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blchm\" (UniqueName: \"kubernetes.io/projected/a34e6738-b233-4ba6-aeb7-f301cedca59a-kube-api-access-blchm\") pod \"a34e6738-b233-4ba6-aeb7-f301cedca59a\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") "
Sep 11 18:00:03.229041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.228918    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-util\") pod \"a34e6738-b233-4ba6-aeb7-f301cedca59a\" (UID: \"a34e6738-b233-4ba6-aeb7-f301cedca59a\") "
Sep 11 18:00:03.229637 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.229609    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-bundle" (OuterVolumeSpecName: "bundle") pod "a34e6738-b233-4ba6-aeb7-f301cedca59a" (UID: "a34e6738-b233-4ba6-aeb7-f301cedca59a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:03.230979 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.230959    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34e6738-b233-4ba6-aeb7-f301cedca59a-kube-api-access-blchm" (OuterVolumeSpecName: "kube-api-access-blchm") pod "a34e6738-b233-4ba6-aeb7-f301cedca59a" (UID: "a34e6738-b233-4ba6-aeb7-f301cedca59a"). InnerVolumeSpecName "kube-api-access-blchm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:03.237127 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.237085    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-util" (OuterVolumeSpecName: "util") pod "a34e6738-b233-4ba6-aeb7-f301cedca59a" (UID: "a34e6738-b233-4ba6-aeb7-f301cedca59a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:03.329927 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.329855    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-bundle\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:03.329927 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.329880    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blchm\" (UniqueName: \"kubernetes.io/projected/a34e6738-b233-4ba6-aeb7-f301cedca59a-kube-api-access-blchm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:03.329927 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:03.329889    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a34e6738-b233-4ba6-aeb7-f301cedca59a-util\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:04.070830 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:04.070801    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd"
Sep 11 18:00:04.070992 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:04.070795    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzkbnd" event={"ID":"a34e6738-b233-4ba6-aeb7-f301cedca59a","Type":"ContainerDied","Data":"d01c09c7e893deff68ed8f63834c37aeb19433c767bd8355bc35ca0f4531dd30"}
Sep 11 18:00:04.070992 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:04.070918    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d01c09c7e893deff68ed8f63834c37aeb19433c767bd8355bc35ca0f4531dd30"
Sep 11 18:00:10.905466 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.905431    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"]
Sep 11 18:00:10.905915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.905668    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerName="util"
Sep 11 18:00:10.905915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.905678    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerName="util"
Sep 11 18:00:10.905915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.905684    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerName="extract"
Sep 11 18:00:10.905915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.905691    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerName="extract"
Sep 11 18:00:10.905915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.905702    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerName="pull"
Sep 11 18:00:10.905915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.905707    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerName="pull"
Sep 11 18:00:10.905915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.905779    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34e6738-b233-4ba6-aeb7-f301cedca59a" containerName="extract"
Sep 11 18:00:10.909779 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.909762    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:10.913259 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.913235    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 18:00:10.913386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.913278    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 18:00:10.914825 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.914805    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p767k\""
Sep 11 18:00:10.922176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.922155    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"]
Sep 11 18:00:10.986318 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.986277    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:10.986447 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.986323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:10.986447 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:10.986380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qscs5\" (UniqueName: \"kubernetes.io/projected/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-kube-api-access-qscs5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:11.087507 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:11.087474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qscs5\" (UniqueName: \"kubernetes.io/projected/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-kube-api-access-qscs5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:11.087676 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:11.087531    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:11.087676 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:11.087561    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:11.087935 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:11.087915    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:11.088004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:11.087945    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:11.096462 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:11.096437    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qscs5\" (UniqueName: \"kubernetes.io/projected/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-kube-api-access-qscs5\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:11.218708 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:11.218627    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:11.493945 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:11.493872    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"]
Sep 11 18:00:11.497278 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:00:11.497251    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5eb10e_9594_45e6_8ec3_2bbc14f3a533.slice/crio-3701b612d355849b7516a15dd3bb7cd139f7dda2f5ab1e43d4e13d3015114579 WatchSource:0}: Error finding container 3701b612d355849b7516a15dd3bb7cd139f7dda2f5ab1e43d4e13d3015114579: Status 404 returned error can't find the container with id 3701b612d355849b7516a15dd3bb7cd139f7dda2f5ab1e43d4e13d3015114579
Sep 11 18:00:12.105374 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.105340    2571 generic.go:356] "Generic (PLEG): container finished" podID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerID="851462171f0d40dfb258b00c6ca29329dd779f79c1d81a5f3be334dd1726854e" exitCode=0
Sep 11 18:00:12.105779 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.105388    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q" event={"ID":"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533","Type":"ContainerDied","Data":"851462171f0d40dfb258b00c6ca29329dd779f79c1d81a5f3be334dd1726854e"}
Sep 11 18:00:12.105779 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.105479    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q" event={"ID":"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533","Type":"ContainerStarted","Data":"3701b612d355849b7516a15dd3bb7cd139f7dda2f5ab1e43d4e13d3015114579"}
Sep 11 18:00:12.641189 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.641160    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"]
Sep 11 18:00:12.644661 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.644564    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.647903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.647736    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:00:12.648128 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.648105    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:00:12.648226 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.648205    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-srmfv\""
Sep 11 18:00:12.648520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.648333    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:00:12.649143 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.649124    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:00:12.667153 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.667091    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"]
Sep 11 18:00:12.699385 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.699364    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba5fc8b-d00b-4dfd-be31-057d612f1539-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.699475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.699398    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aba5fc8b-d00b-4dfd-be31-057d612f1539-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.699475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.699423    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69h4\" (UniqueName: \"kubernetes.io/projected/aba5fc8b-d00b-4dfd-be31-057d612f1539-kube-api-access-v69h4\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.699563 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.699484    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aba5fc8b-d00b-4dfd-be31-057d612f1539-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.699563 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.699532    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aba5fc8b-d00b-4dfd-be31-057d612f1539-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.800355 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.800321    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aba5fc8b-d00b-4dfd-be31-057d612f1539-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.800521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.800374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aba5fc8b-d00b-4dfd-be31-057d612f1539-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.800521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.800411    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba5fc8b-d00b-4dfd-be31-057d612f1539-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.800521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.800429    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aba5fc8b-d00b-4dfd-be31-057d612f1539-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.800521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.800449    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69h4\" (UniqueName: \"kubernetes.io/projected/aba5fc8b-d00b-4dfd-be31-057d612f1539-kube-api-access-v69h4\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.800782 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.800759    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aba5fc8b-d00b-4dfd-be31-057d612f1539-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.802617 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.802591    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba5fc8b-d00b-4dfd-be31-057d612f1539-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.802838 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.802817    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aba5fc8b-d00b-4dfd-be31-057d612f1539-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.802882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.802853    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aba5fc8b-d00b-4dfd-be31-057d612f1539-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.809544 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.809522    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69h4\" (UniqueName: \"kubernetes.io/projected/aba5fc8b-d00b-4dfd-be31-057d612f1539-kube-api-access-v69h4\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-fljr7\" (UID: \"aba5fc8b-d00b-4dfd-be31-057d612f1539\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:12.955796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:12.955765    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:13.111402 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:13.111366    2571 generic.go:356] "Generic (PLEG): container finished" podID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerID="d577a2f1bae1a7c95697934199d7aea095341aa724992d80c14780d908ff1259" exitCode=0
Sep 11 18:00:13.111737 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:13.111443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q" event={"ID":"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533","Type":"ContainerDied","Data":"d577a2f1bae1a7c95697934199d7aea095341aa724992d80c14780d908ff1259"}
Sep 11 18:00:13.231734 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:13.231168    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"]
Sep 11 18:00:13.233440 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:00:13.233416    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba5fc8b_d00b_4dfd_be31_057d612f1539.slice/crio-76fb5fa76a8dea87246fe745ba50e20e9bc83c82e884f4c624331cb4849138f5 WatchSource:0}: Error finding container 76fb5fa76a8dea87246fe745ba50e20e9bc83c82e884f4c624331cb4849138f5: Status 404 returned error can't find the container with id 76fb5fa76a8dea87246fe745ba50e20e9bc83c82e884f4c624331cb4849138f5
Sep 11 18:00:14.118919 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:14.118882    2571 generic.go:356] "Generic (PLEG): container finished" podID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerID="681c6a8d4cafedda0e2d63670e3f41e870e1439e83b5327cf092181c0bfd461a" exitCode=0
Sep 11 18:00:14.119372 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:14.118972    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q" event={"ID":"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533","Type":"ContainerDied","Data":"681c6a8d4cafedda0e2d63670e3f41e870e1439e83b5327cf092181c0bfd461a"}
Sep 11 18:00:14.120324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:14.120284    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7" event={"ID":"aba5fc8b-d00b-4dfd-be31-057d612f1539","Type":"ContainerStarted","Data":"76fb5fa76a8dea87246fe745ba50e20e9bc83c82e884f4c624331cb4849138f5"}
Sep 11 18:00:16.728332 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.728293    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:16.828061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.828040    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-bundle\") pod \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") "
Sep 11 18:00:16.828133 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.828104    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qscs5\" (UniqueName: \"kubernetes.io/projected/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-kube-api-access-qscs5\") pod \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") "
Sep 11 18:00:16.828178 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.828131    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-util\") pod \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\" (UID: \"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533\") "
Sep 11 18:00:16.828828 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.828804    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-bundle" (OuterVolumeSpecName: "bundle") pod "8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" (UID: "8a5eb10e-9594-45e6-8ec3-2bbc14f3a533"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:16.830018 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.829997    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-kube-api-access-qscs5" (OuterVolumeSpecName: "kube-api-access-qscs5") pod "8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" (UID: "8a5eb10e-9594-45e6-8ec3-2bbc14f3a533"). InnerVolumeSpecName "kube-api-access-qscs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:16.858046 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.858015    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-util" (OuterVolumeSpecName: "util") pod "8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" (UID: "8a5eb10e-9594-45e6-8ec3-2bbc14f3a533"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:16.929333 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.929285    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-bundle\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:16.929422 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.929336    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qscs5\" (UniqueName: \"kubernetes.io/projected/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-kube-api-access-qscs5\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:16.929422 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:16.929350    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a5eb10e-9594-45e6-8ec3-2bbc14f3a533-util\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:17.135646 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:17.135561    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7" event={"ID":"aba5fc8b-d00b-4dfd-be31-057d612f1539","Type":"ContainerStarted","Data":"a2196da179cc3a8cf2b44c510f425c5fe857aab19bc19049cfa467c96635f7c4"}
Sep 11 18:00:17.137392 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:17.137365    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q" event={"ID":"8a5eb10e-9594-45e6-8ec3-2bbc14f3a533","Type":"ContainerDied","Data":"3701b612d355849b7516a15dd3bb7cd139f7dda2f5ab1e43d4e13d3015114579"}
Sep 11 18:00:17.137515 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:17.137398    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0ghw8q"
Sep 11 18:00:17.137515 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:17.137401    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3701b612d355849b7516a15dd3bb7cd139f7dda2f5ab1e43d4e13d3015114579"
Sep 11 18:00:18.142732 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:18.142639    2571 generic.go:356] "Generic (PLEG): container finished" podID="aba5fc8b-d00b-4dfd-be31-057d612f1539" containerID="a2196da179cc3a8cf2b44c510f425c5fe857aab19bc19049cfa467c96635f7c4" exitCode=0
Sep 11 18:00:18.142732 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:18.142691    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7" event={"ID":"aba5fc8b-d00b-4dfd-be31-057d612f1539","Type":"ContainerDied","Data":"a2196da179cc3a8cf2b44c510f425c5fe857aab19bc19049cfa467c96635f7c4"}
Sep 11 18:00:19.149072 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:19.149040    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7" event={"ID":"aba5fc8b-d00b-4dfd-be31-057d612f1539","Type":"ContainerStarted","Data":"84233879f573f99d0c34917a42064a9720d1b3c12a9d8a78814467db04ab5df2"}
Sep 11 18:00:19.149488 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:19.149177    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:30.154856 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:30.154825    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7"
Sep 11 18:00:30.179496 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:30.179443    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-fljr7" podStartSLOduration=13.269132378 podStartE2EDuration="18.179428698s" totalImagesPullingTime="3.888319275s" totalInitContainerRuntime="1.021977045s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:19.200359127 +0000 UTC m=+382.970425475" watchObservedRunningTime="2026-09-11 18:00:30.179428698 +0000 UTC m=+393.949495047"
Sep 11 18:00:33.628727 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.628693    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"]
Sep 11 18:00:33.629096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.628941    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerName="util"
Sep 11 18:00:33.629096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.628953    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerName="util"
Sep 11 18:00:33.629096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.628962    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerName="pull"
Sep 11 18:00:33.629096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.628967    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerName="pull"
Sep 11 18:00:33.629096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.628976    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerName="extract"
Sep 11 18:00:33.629096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.628982    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerName="extract"
Sep 11 18:00:33.629096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.629035    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a5eb10e-9594-45e6-8ec3-2bbc14f3a533" containerName="extract"
Sep 11 18:00:33.637214 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.637192    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.642027 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.642007    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mjp97\""
Sep 11 18:00:33.643260 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.643203    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:00:33.643260 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.643252    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 18:00:33.643433 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.643339    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:00:33.643521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.643502    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 18:00:33.658956 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.658934    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"]
Sep 11 18:00:33.752505 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.752473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7psm2\" (UniqueName: \"kubernetes.io/projected/a3112d22-7697-4c4e-b898-0c8093d579cd-kube-api-access-7psm2\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.752659 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.752520    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3112d22-7697-4c4e-b898-0c8093d579cd-tls-certs\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.752659 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.752568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3112d22-7697-4c4e-b898-0c8093d579cd-tmp\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.852879 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.852849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7psm2\" (UniqueName: \"kubernetes.io/projected/a3112d22-7697-4c4e-b898-0c8093d579cd-kube-api-access-7psm2\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.853016 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.852888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3112d22-7697-4c4e-b898-0c8093d579cd-tls-certs\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.853016 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.852903    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3112d22-7697-4c4e-b898-0c8093d579cd-tmp\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.855174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.855154    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3112d22-7697-4c4e-b898-0c8093d579cd-tmp\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.855541 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.855521    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3112d22-7697-4c4e-b898-0c8093d579cd-tls-certs\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.867494 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.867471    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7psm2\" (UniqueName: \"kubernetes.io/projected/a3112d22-7697-4c4e-b898-0c8093d579cd-kube-api-access-7psm2\") pod \"kube-auth-proxy-7ff57cc6b9-r4f9g\" (UID: \"a3112d22-7697-4c4e-b898-0c8093d579cd\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:33.946113 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:33.946093    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"
Sep 11 18:00:34.231145 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:34.231122    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g"]
Sep 11 18:00:34.237821 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:00:34.237783    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3112d22_7697_4c4e_b898_0c8093d579cd.slice/crio-cfd90987192342bca0e2f5e55ad1b6662220e5ef273cffb010fdde39bc379cf8 WatchSource:0}: Error finding container cfd90987192342bca0e2f5e55ad1b6662220e5ef273cffb010fdde39bc379cf8: Status 404 returned error can't find the container with id cfd90987192342bca0e2f5e55ad1b6662220e5ef273cffb010fdde39bc379cf8
Sep 11 18:00:35.185032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.184967    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"]
Sep 11 18:00:35.189103 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.188743    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.192159 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.191922    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 18:00:35.193392 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.192582    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 18:00:35.193392 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.192601    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zsf55\""
Sep 11 18:00:35.193392 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.192995    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 18:00:35.193392 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.193179    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 18:00:35.215156 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.215125    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g" event={"ID":"a3112d22-7697-4c4e-b898-0c8093d579cd","Type":"ContainerStarted","Data":"cfd90987192342bca0e2f5e55ad1b6662220e5ef273cffb010fdde39bc379cf8"}
Sep 11 18:00:35.222475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.222415    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"]
Sep 11 18:00:35.265449 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.265419    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.265579 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.265463    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.265579 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.265497    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42a108d-d283-40b4-b133-9bcab1b013f1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.265692 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.265598    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.265692 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.265629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg8rf\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-kube-api-access-hg8rf\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.265692 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.265680    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.265845 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.265716    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.366428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.366388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.366615 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.366448    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.366615 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.366474    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.366615 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.366503    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42a108d-d283-40b4-b133-9bcab1b013f1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.366615 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.366549    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.366956 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.366914    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg8rf\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-kube-api-access-hg8rf\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.367061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.366991    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.368454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.368354    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.369691 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.369567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.369819 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.369797    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42a108d-d283-40b4-b133-9bcab1b013f1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.370321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.369967    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.370321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.369971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.382196 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.382166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.384581 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.384558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg8rf\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-kube-api-access-hg8rf\") pod \"istiod-openshift-gateway-7b7995d5cd-wph6l\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.501561 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.501461    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:35.790390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:35.790362    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"]
Sep 11 18:00:36.879354 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:00:36.879323    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42a108d_d283_40b4_b133_9bcab1b013f1.slice/crio-ac455a5d2ae4b90c653c996a7276bfedb653c4f96abf7bd80d4fa1bea9696a56 WatchSource:0}: Error finding container ac455a5d2ae4b90c653c996a7276bfedb653c4f96abf7bd80d4fa1bea9696a56: Status 404 returned error can't find the container with id ac455a5d2ae4b90c653c996a7276bfedb653c4f96abf7bd80d4fa1bea9696a56
Sep 11 18:00:37.224552 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:37.224515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g" event={"ID":"a3112d22-7697-4c4e-b898-0c8093d579cd","Type":"ContainerStarted","Data":"4ff6c8b8675947a7c524854ac3556ef7c83c9f4a6c85c76780c722eaa5fb2fc5"}
Sep 11 18:00:37.225760 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:37.225735    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l" event={"ID":"f42a108d-d283-40b4-b133-9bcab1b013f1","Type":"ContainerStarted","Data":"ac455a5d2ae4b90c653c996a7276bfedb653c4f96abf7bd80d4fa1bea9696a56"}
Sep 11 18:00:37.245124 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:37.245078    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-r4f9g" podStartSLOduration=1.546802612 podStartE2EDuration="4.245063395s" totalImagesPullingTime="2.698260783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:37.242888749 +0000 UTC m=+401.012955097" watchObservedRunningTime="2026-09-11 18:00:37.245063395 +0000 UTC m=+401.015129743"
Sep 11 18:00:42.263942 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:42.263903    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:00:42.264337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:42.263987    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:00:43.254898 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:43.254855    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l" event={"ID":"f42a108d-d283-40b4-b133-9bcab1b013f1","Type":"ContainerStarted","Data":"c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd"}
Sep 11 18:00:43.256049 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:43.256020    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:43.258851 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:43.258401    2571 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-wph6l container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:00:43.258851 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:43.258456    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l" podUID="f42a108d-d283-40b4-b133-9bcab1b013f1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:00:43.277873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:43.277822    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l" podStartSLOduration=2.895848957 podStartE2EDuration="8.277805253s" totalImagesPullingTime="5.381956296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:43.276396461 +0000 UTC m=+407.046462801" watchObservedRunningTime="2026-09-11 18:00:43.277805253 +0000 UTC m=+407.047871611"
Sep 11 18:00:44.036165 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.036131    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zw2cp"]
Sep 11 18:00:44.039415 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.039394    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:44.043264 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.043242    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 18:00:44.043367 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.043268    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fxrkq\""
Sep 11 18:00:44.056017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.055994    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zw2cp"]
Sep 11 18:00:44.131531 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.131500    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c79da3-d8b5-4839-810d-66ce64aef094-cert\") pod \"odh-model-controller-7998c67dc4-zw2cp\" (UID: \"c3c79da3-d8b5-4839-810d-66ce64aef094\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:44.131684 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.131551    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxj8j\" (UniqueName: \"kubernetes.io/projected/c3c79da3-d8b5-4839-810d-66ce64aef094-kube-api-access-bxj8j\") pod \"odh-model-controller-7998c67dc4-zw2cp\" (UID: \"c3c79da3-d8b5-4839-810d-66ce64aef094\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:44.232866 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.232831    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c79da3-d8b5-4839-810d-66ce64aef094-cert\") pod \"odh-model-controller-7998c67dc4-zw2cp\" (UID: \"c3c79da3-d8b5-4839-810d-66ce64aef094\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:44.232866 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.232874    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxj8j\" (UniqueName: \"kubernetes.io/projected/c3c79da3-d8b5-4839-810d-66ce64aef094-kube-api-access-bxj8j\") pod \"odh-model-controller-7998c67dc4-zw2cp\" (UID: \"c3c79da3-d8b5-4839-810d-66ce64aef094\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:44.233062 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:00:44.232997    2571 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 18:00:44.233062 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:00:44.233055    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3c79da3-d8b5-4839-810d-66ce64aef094-cert podName:c3c79da3-d8b5-4839-810d-66ce64aef094 nodeName:}" failed. No retries permitted until 2026-09-11 18:00:44.733038798 +0000 UTC m=+408.503105143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c3c79da3-d8b5-4839-810d-66ce64aef094-cert") pod "odh-model-controller-7998c67dc4-zw2cp" (UID: "c3c79da3-d8b5-4839-810d-66ce64aef094") : secret "odh-model-controller-webhook-cert" not found
Sep 11 18:00:44.241626 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.241602    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxj8j\" (UniqueName: \"kubernetes.io/projected/c3c79da3-d8b5-4839-810d-66ce64aef094-kube-api-access-bxj8j\") pod \"odh-model-controller-7998c67dc4-zw2cp\" (UID: \"c3c79da3-d8b5-4839-810d-66ce64aef094\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:44.259893 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.259871    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:44.736584 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.736554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c79da3-d8b5-4839-810d-66ce64aef094-cert\") pod \"odh-model-controller-7998c67dc4-zw2cp\" (UID: \"c3c79da3-d8b5-4839-810d-66ce64aef094\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:44.738793 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.738774    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3c79da3-d8b5-4839-810d-66ce64aef094-cert\") pod \"odh-model-controller-7998c67dc4-zw2cp\" (UID: \"c3c79da3-d8b5-4839-810d-66ce64aef094\") " pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:44.833139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.833112    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-thghb"]
Sep 11 18:00:44.836468 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.836452    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:44.839371 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.839350    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 18:00:44.839371 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.839368    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-s5jcr\""
Sep 11 18:00:44.846795 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.846773    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-thghb"]
Sep 11 18:00:44.937898 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.937870    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfv4m\" (UniqueName: \"kubernetes.io/projected/a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43-kube-api-access-xfv4m\") pod \"model-serving-api-54d89bd5fc-thghb\" (UID: \"a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43\") " pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:44.938054 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.937961    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43-tls-certs\") pod \"model-serving-api-54d89bd5fc-thghb\" (UID: \"a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43\") " pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:44.949016 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:44.948990    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:45.038804 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:45.038774    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43-tls-certs\") pod \"model-serving-api-54d89bd5fc-thghb\" (UID: \"a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43\") " pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:45.038932 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:45.038814    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfv4m\" (UniqueName: \"kubernetes.io/projected/a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43-kube-api-access-xfv4m\") pod \"model-serving-api-54d89bd5fc-thghb\" (UID: \"a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43\") " pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:45.041163 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:45.041140    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43-tls-certs\") pod \"model-serving-api-54d89bd5fc-thghb\" (UID: \"a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43\") " pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:45.047676 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:45.047649    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfv4m\" (UniqueName: \"kubernetes.io/projected/a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43-kube-api-access-xfv4m\") pod \"model-serving-api-54d89bd5fc-thghb\" (UID: \"a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43\") " pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:45.147291 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:45.147257    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:45.223762 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:45.222944    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-zw2cp"]
Sep 11 18:00:45.263979 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:45.263915    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp" event={"ID":"c3c79da3-d8b5-4839-810d-66ce64aef094","Type":"ContainerStarted","Data":"0fa846935eb585e62b34262c92373797f7ae5ea185c6f7ed8ea2e2969be3bce3"}
Sep 11 18:00:45.414486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:45.414461    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-thghb"]
Sep 11 18:00:45.415951 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:00:45.415928    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a91f0f_8a3f_44b8_b8fb_7820efbb5a43.slice/crio-e92676806a80823cde5a281a0fe43a0539ebda987355946b52aa1c70995359fc WatchSource:0}: Error finding container e92676806a80823cde5a281a0fe43a0539ebda987355946b52aa1c70995359fc: Status 404 returned error can't find the container with id e92676806a80823cde5a281a0fe43a0539ebda987355946b52aa1c70995359fc
Sep 11 18:00:46.271545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:46.271506    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-thghb" event={"ID":"a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43","Type":"ContainerStarted","Data":"e92676806a80823cde5a281a0fe43a0539ebda987355946b52aa1c70995359fc"}
Sep 11 18:00:49.286881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:49.286840    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-thghb" event={"ID":"a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43","Type":"ContainerStarted","Data":"a83790c9f66a60011ef984d6e0ef499d6ba8fe140a9406ccbd55e4a78edd3965"}
Sep 11 18:00:49.287327 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:49.286961    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:00:49.288491 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:49.288470    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp" event={"ID":"c3c79da3-d8b5-4839-810d-66ce64aef094","Type":"ContainerStarted","Data":"c6ed1aa7f1e1ee822d2889c16f45b290cc86255e241610e50cbd3b1dde788098"}
Sep 11 18:00:49.288596 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:49.288522    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:00:49.327864 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:49.327816    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-thghb" podStartSLOduration=2.38247791 podStartE2EDuration="5.327800017s" totalImagesPullingTime="2.945322107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:49.325004739 +0000 UTC m=+413.095071079" watchObservedRunningTime="2026-09-11 18:00:49.327800017 +0000 UTC m=+413.097866365"
Sep 11 18:00:49.351721 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:49.351681    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp" podStartSLOduration=2.168066009 podStartE2EDuration="5.351668003s" totalImagesPullingTime="3.183601994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:49.351027903 +0000 UTC m=+413.121094250" watchObservedRunningTime="2026-09-11 18:00:49.351668003 +0000 UTC m=+413.121734351"
Sep 11 18:00:54.902717 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.902684    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"]
Sep 11 18:00:54.909048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.909029    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:54.912139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.912120    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 18:00:54.912267 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.912120    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-cxlh7\""
Sep 11 18:00:54.927277 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.927253    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"]
Sep 11 18:00:54.929917 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.929904    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:54.933688 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.933654    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4fw68\""
Sep 11 18:00:54.933796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.933690    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 18:00:54.935065 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.935044    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"]
Sep 11 18:00:54.949717 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:54.949694    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"]
Sep 11 18:00:55.022038 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.022002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shlq\" (UniqueName: \"kubernetes.io/projected/c959416b-dbbf-456e-83e6-b03b2a689e0b-kube-api-access-2shlq\") pod \"llmisvc-controller-manager-777c6bd7d5-66zsz\" (UID: \"c959416b-dbbf-456e-83e6-b03b2a689e0b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:55.022038 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.022040    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c959416b-dbbf-456e-83e6-b03b2a689e0b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66zsz\" (UID: \"c959416b-dbbf-456e-83e6-b03b2a689e0b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:55.022256 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.022069    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qtg\" (UniqueName: \"kubernetes.io/projected/edaf25f5-8c33-433d-b05c-1fb1b04bca25-kube-api-access-z7qtg\") pod \"kserve-controller-manager-d6f4b65b8-nkr9j\" (UID: \"edaf25f5-8c33-433d-b05c-1fb1b04bca25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:55.022256 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.022140    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edaf25f5-8c33-433d-b05c-1fb1b04bca25-cert\") pod \"kserve-controller-manager-d6f4b65b8-nkr9j\" (UID: \"edaf25f5-8c33-433d-b05c-1fb1b04bca25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:55.122724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.122691    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qtg\" (UniqueName: \"kubernetes.io/projected/edaf25f5-8c33-433d-b05c-1fb1b04bca25-kube-api-access-z7qtg\") pod \"kserve-controller-manager-d6f4b65b8-nkr9j\" (UID: \"edaf25f5-8c33-433d-b05c-1fb1b04bca25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:55.122724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.122729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edaf25f5-8c33-433d-b05c-1fb1b04bca25-cert\") pod \"kserve-controller-manager-d6f4b65b8-nkr9j\" (UID: \"edaf25f5-8c33-433d-b05c-1fb1b04bca25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:55.122963 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.122806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shlq\" (UniqueName: \"kubernetes.io/projected/c959416b-dbbf-456e-83e6-b03b2a689e0b-kube-api-access-2shlq\") pod \"llmisvc-controller-manager-777c6bd7d5-66zsz\" (UID: \"c959416b-dbbf-456e-83e6-b03b2a689e0b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:55.122963 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.122829    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c959416b-dbbf-456e-83e6-b03b2a689e0b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66zsz\" (UID: \"c959416b-dbbf-456e-83e6-b03b2a689e0b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:55.122963 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:00:55.122917    2571 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 11 18:00:55.123112 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:00:55.122915    2571 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 18:00:55.123112 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:00:55.122998    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c959416b-dbbf-456e-83e6-b03b2a689e0b-cert podName:c959416b-dbbf-456e-83e6-b03b2a689e0b nodeName:}" failed. No retries permitted until 2026-09-11 18:00:55.622976193 +0000 UTC m=+419.393042521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c959416b-dbbf-456e-83e6-b03b2a689e0b-cert") pod "llmisvc-controller-manager-777c6bd7d5-66zsz" (UID: "c959416b-dbbf-456e-83e6-b03b2a689e0b") : secret "llmisvc-webhook-server-cert" not found
Sep 11 18:00:55.123112 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:00:55.123052    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edaf25f5-8c33-433d-b05c-1fb1b04bca25-cert podName:edaf25f5-8c33-433d-b05c-1fb1b04bca25 nodeName:}" failed. No retries permitted until 2026-09-11 18:00:55.623031909 +0000 UTC m=+419.393098248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/edaf25f5-8c33-433d-b05c-1fb1b04bca25-cert") pod "kserve-controller-manager-d6f4b65b8-nkr9j" (UID: "edaf25f5-8c33-433d-b05c-1fb1b04bca25") : secret "kserve-webhook-server-cert" not found
Sep 11 18:00:55.138987 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.138958    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shlq\" (UniqueName: \"kubernetes.io/projected/c959416b-dbbf-456e-83e6-b03b2a689e0b-kube-api-access-2shlq\") pod \"llmisvc-controller-manager-777c6bd7d5-66zsz\" (UID: \"c959416b-dbbf-456e-83e6-b03b2a689e0b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:55.145151 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.145127    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qtg\" (UniqueName: \"kubernetes.io/projected/edaf25f5-8c33-433d-b05c-1fb1b04bca25-kube-api-access-z7qtg\") pod \"kserve-controller-manager-d6f4b65b8-nkr9j\" (UID: \"edaf25f5-8c33-433d-b05c-1fb1b04bca25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:55.626360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.626285    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c959416b-dbbf-456e-83e6-b03b2a689e0b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66zsz\" (UID: \"c959416b-dbbf-456e-83e6-b03b2a689e0b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:55.626565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.626376    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edaf25f5-8c33-433d-b05c-1fb1b04bca25-cert\") pod \"kserve-controller-manager-d6f4b65b8-nkr9j\" (UID: \"edaf25f5-8c33-433d-b05c-1fb1b04bca25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:55.628663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.628638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edaf25f5-8c33-433d-b05c-1fb1b04bca25-cert\") pod \"kserve-controller-manager-d6f4b65b8-nkr9j\" (UID: \"edaf25f5-8c33-433d-b05c-1fb1b04bca25\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:55.628768 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.628682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c959416b-dbbf-456e-83e6-b03b2a689e0b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66zsz\" (UID: \"c959416b-dbbf-456e-83e6-b03b2a689e0b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:55.819195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.819168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:55.840152 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:55.840123    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:00:56.104014 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:00:56.103985    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedaf25f5_8c33_433d_b05c_1fb1b04bca25.slice/crio-7ec71599ed74e1b28d55d57d734ce479f0093d7d090adad8c83de7036c975ca4 WatchSource:0}: Error finding container 7ec71599ed74e1b28d55d57d734ce479f0093d7d090adad8c83de7036c975ca4: Status 404 returned error can't find the container with id 7ec71599ed74e1b28d55d57d734ce479f0093d7d090adad8c83de7036c975ca4
Sep 11 18:00:56.105167 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.105143    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"]
Sep 11 18:00:56.130995 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.130971    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"]
Sep 11 18:00:56.133037 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:00:56.133007    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc959416b_dbbf_456e_83e6_b03b2a689e0b.slice/crio-44d49aa9cc40b89c5031ab9e4ae4b4a914d6bb36497f6d6cae24048843ef96c1 WatchSource:0}: Error finding container 44d49aa9cc40b89c5031ab9e4ae4b4a914d6bb36497f6d6cae24048843ef96c1: Status 404 returned error can't find the container with id 44d49aa9cc40b89c5031ab9e4ae4b4a914d6bb36497f6d6cae24048843ef96c1
Sep 11 18:00:56.318609 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.318570    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz" event={"ID":"c959416b-dbbf-456e-83e6-b03b2a689e0b","Type":"ContainerStarted","Data":"44d49aa9cc40b89c5031ab9e4ae4b4a914d6bb36497f6d6cae24048843ef96c1"}
Sep 11 18:00:56.319706 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.319685    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j" event={"ID":"edaf25f5-8c33-433d-b05c-1fb1b04bca25","Type":"ContainerStarted","Data":"7ec71599ed74e1b28d55d57d734ce479f0093d7d090adad8c83de7036c975ca4"}
Sep 11 18:00:56.461100 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.461063    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"]
Sep 11 18:00:56.461282 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.461261    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l" podUID="f42a108d-d283-40b4-b133-9bcab1b013f1" containerName="discovery" containerID="cri-o://c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd" gracePeriod=30
Sep 11 18:00:56.702453 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.702430    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:56.836190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.836107    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-dns-cert\") pod \"f42a108d-d283-40b4-b133-9bcab1b013f1\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") "
Sep 11 18:00:56.836190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.836138    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-cacerts\") pod \"f42a108d-d283-40b4-b133-9bcab1b013f1\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") "
Sep 11 18:00:56.836190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.836165    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg8rf\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-kube-api-access-hg8rf\") pod \"f42a108d-d283-40b4-b133-9bcab1b013f1\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") "
Sep 11 18:00:56.836190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.836185    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42a108d-d283-40b4-b133-9bcab1b013f1-local-certs\") pod \"f42a108d-d283-40b4-b133-9bcab1b013f1\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") "
Sep 11 18:00:56.836561 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.836329    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-ca-configmap\") pod \"f42a108d-d283-40b4-b133-9bcab1b013f1\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") "
Sep 11 18:00:56.836561 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.836389    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-token\") pod \"f42a108d-d283-40b4-b133-9bcab1b013f1\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") "
Sep 11 18:00:56.836561 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.836417    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-kubeconfig\") pod \"f42a108d-d283-40b4-b133-9bcab1b013f1\" (UID: \"f42a108d-d283-40b4-b133-9bcab1b013f1\") "
Sep 11 18:00:56.836729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.836701    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f42a108d-d283-40b4-b133-9bcab1b013f1" (UID: "f42a108d-d283-40b4-b133-9bcab1b013f1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:00:56.838642 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.838617    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-kube-api-access-hg8rf" (OuterVolumeSpecName: "kube-api-access-hg8rf") pod "f42a108d-d283-40b4-b133-9bcab1b013f1" (UID: "f42a108d-d283-40b4-b133-9bcab1b013f1"). InnerVolumeSpecName "kube-api-access-hg8rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:56.838751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.838719    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f42a108d-d283-40b4-b133-9bcab1b013f1-local-certs" (OuterVolumeSpecName: "local-certs") pod "f42a108d-d283-40b4-b133-9bcab1b013f1" (UID: "f42a108d-d283-40b4-b133-9bcab1b013f1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:00:56.838818 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.838742    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-cacerts" (OuterVolumeSpecName: "cacerts") pod "f42a108d-d283-40b4-b133-9bcab1b013f1" (UID: "f42a108d-d283-40b4-b133-9bcab1b013f1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:56.838818 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.838801    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f42a108d-d283-40b4-b133-9bcab1b013f1" (UID: "f42a108d-d283-40b4-b133-9bcab1b013f1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:56.838920 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.838862    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f42a108d-d283-40b4-b133-9bcab1b013f1" (UID: "f42a108d-d283-40b4-b133-9bcab1b013f1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:00:56.838975 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.838957    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "f42a108d-d283-40b4-b133-9bcab1b013f1" (UID: "f42a108d-d283-40b4-b133-9bcab1b013f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:00:56.937938 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.937910    2571 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-dns-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:56.937938 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.937934    2571 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-cacerts\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:56.937938 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.937944    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg8rf\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-kube-api-access-hg8rf\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:56.938138 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.937953    2571 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f42a108d-d283-40b4-b133-9bcab1b013f1-local-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:56.938138 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.937961    2571 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-csr-ca-configmap\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:56.938138 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.937969    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:56.938138 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:56.937979    2571 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f42a108d-d283-40b4-b133-9bcab1b013f1-istio-kubeconfig\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:00:57.326787 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.326751    2571 generic.go:356] "Generic (PLEG): container finished" podID="f42a108d-d283-40b4-b133-9bcab1b013f1" containerID="c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd" exitCode=0
Sep 11 18:00:57.327257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.326801    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"
Sep 11 18:00:57.327257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.326834    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l" event={"ID":"f42a108d-d283-40b4-b133-9bcab1b013f1","Type":"ContainerDied","Data":"c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd"}
Sep 11 18:00:57.327257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.326888    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l" event={"ID":"f42a108d-d283-40b4-b133-9bcab1b013f1","Type":"ContainerDied","Data":"ac455a5d2ae4b90c653c996a7276bfedb653c4f96abf7bd80d4fa1bea9696a56"}
Sep 11 18:00:57.327257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.326916    2571 scope.go:117] "RemoveContainer" containerID="c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd"
Sep 11 18:00:57.336768 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.336749    2571 scope.go:117] "RemoveContainer" containerID="c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd"
Sep 11 18:00:57.337036 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:00:57.337018    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd\": container with ID starting with c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd not found: ID does not exist" containerID="c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd"
Sep 11 18:00:57.337090 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.337044    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd"} err="failed to get container status \"c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd\": rpc error: code = NotFound desc = could not find container \"c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd\": container with ID starting with c83d3a85897056b6f5631b90418c5251d3c1835974f4094cb7a5e20fd32eb4cd not found: ID does not exist"
Sep 11 18:00:57.352593 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.352566    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"]
Sep 11 18:00:57.360013 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:57.359991    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wph6l"]
Sep 11 18:00:58.732761 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:58.732730    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f42a108d-d283-40b4-b133-9bcab1b013f1" path="/var/lib/kubelet/pods/f42a108d-d283-40b4-b133-9bcab1b013f1/volumes"
Sep 11 18:00:59.341119 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:59.341087    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j" event={"ID":"edaf25f5-8c33-433d-b05c-1fb1b04bca25","Type":"ContainerStarted","Data":"f3aef4ea6955000e0cad141722c924f38d972f68bf40c185fc41580e67788f1b"}
Sep 11 18:00:59.341350 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:59.341186    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:00:59.361215 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:00:59.361162    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j" podStartSLOduration=2.7428740339999997 podStartE2EDuration="5.361146557s" totalImagesPullingTime="2.618272523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:59.359770927 +0000 UTC m=+423.129837269" watchObservedRunningTime="2026-09-11 18:00:59.361146557 +0000 UTC m=+423.131212905"
Sep 11 18:01:00.295089 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:00.295059    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-zw2cp"
Sep 11 18:01:00.297362 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:00.297341    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-thghb"
Sep 11 18:01:01.351886 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:01.351854    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz" event={"ID":"c959416b-dbbf-456e-83e6-b03b2a689e0b","Type":"ContainerStarted","Data":"7e0a6851e8e1e8873ee0e484e6da1772a6ff8a932c8a59e3f47998b46a5b8114"}
Sep 11 18:01:01.352269 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:01.351945    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:01:01.370007 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:01.369961    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz" podStartSLOduration=3.144920227 podStartE2EDuration="7.369944618s" totalImagesPullingTime="4.225024391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:01:01.369215947 +0000 UTC m=+425.139282298" watchObservedRunningTime="2026-09-11 18:01:01.369944618 +0000 UTC m=+425.140010997"
Sep 11 18:01:30.350926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:30.350852    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-nkr9j"
Sep 11 18:01:32.359426 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:32.359396    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66zsz"
Sep 11 18:01:50.022375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.022335    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"]
Sep 11 18:01:50.022849 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.022751    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f42a108d-d283-40b4-b133-9bcab1b013f1" containerName="discovery"
Sep 11 18:01:50.022849 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.022768    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f42a108d-d283-40b4-b133-9bcab1b013f1" containerName="discovery"
Sep 11 18:01:50.022966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.022858    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f42a108d-d283-40b4-b133-9bcab1b013f1" containerName="discovery"
Sep 11 18:01:50.025869 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.025849    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.028815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.028794    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p767k\""
Sep 11 18:01:50.030163 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.030138    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 11 18:01:50.030251 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.030165    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 11 18:01:50.037076 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.037047    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"]
Sep 11 18:01:50.127735 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.127705    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"]
Sep 11 18:01:50.131057 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.131032    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.131207 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.131185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97j5z\" (UniqueName: \"kubernetes.io/projected/898d2f73-ea2b-4a8f-941a-1beb15768f7c-kube-api-access-97j5z\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.131264 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.131248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.131321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.131292    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.141313 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.141272    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"]
Sep 11 18:01:50.232166 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.232139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.232354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.232176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.232354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.232202    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.232354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.232278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.232354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.232340    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4zs7\" (UniqueName: \"kubernetes.io/projected/c8eb91c3-06ad-4385-be14-d061616cca3e-kube-api-access-r4zs7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.232583 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.232371    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97j5z\" (UniqueName: \"kubernetes.io/projected/898d2f73-ea2b-4a8f-941a-1beb15768f7c-kube-api-access-97j5z\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.232583 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.232509    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.232689 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.232646    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.237288 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.237265    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"]
Sep 11 18:01:50.240662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.240648    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.246905 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.246882    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97j5z\" (UniqueName: \"kubernetes.io/projected/898d2f73-ea2b-4a8f-941a-1beb15768f7c-kube-api-access-97j5z\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.249130 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.249108    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"]
Sep 11 18:01:50.333430 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.333345    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.333430 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.333379    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz7hj\" (UniqueName: \"kubernetes.io/projected/8d9ba555-0b35-4a4e-8343-31d911a7d271-kube-api-access-lz7hj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.333430 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.333405    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4zs7\" (UniqueName: \"kubernetes.io/projected/c8eb91c3-06ad-4385-be14-d061616cca3e-kube-api-access-r4zs7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.333711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.333490    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.333711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.333573    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.333711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.333612    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.333850 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.333836    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.333919 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.333902    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.339287 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.339273    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:50.342585 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.342520    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4zs7\" (UniqueName: \"kubernetes.io/projected/c8eb91c3-06ad-4385-be14-d061616cca3e-kube-api-access-r4zs7\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.348140 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.348117    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"]
Sep 11 18:01:50.351791 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.351775    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.367580 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.367542    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"]
Sep 11 18:01:50.434209 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.434179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.434386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.434219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.434386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.434238    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz7hj\" (UniqueName: \"kubernetes.io/projected/8d9ba555-0b35-4a4e-8343-31d911a7d271-kube-api-access-lz7hj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.434386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.434314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmff6\" (UniqueName: \"kubernetes.io/projected/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-kube-api-access-cmff6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.434386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.434354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.434636 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.434429    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.434691 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.434656    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.434734 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.434662    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.440139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.440117    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:50.442586 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.442567    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz7hj\" (UniqueName: \"kubernetes.io/projected/8d9ba555-0b35-4a4e-8343-31d911a7d271-kube-api-access-lz7hj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.535008 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.534971    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.535163 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.535021    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.535163 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.535053    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmff6\" (UniqueName: \"kubernetes.io/projected/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-kube-api-access-cmff6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.535380 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.535363    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.535441 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.535421    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.544062 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.544042    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmff6\" (UniqueName: \"kubernetes.io/projected/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-kube-api-access-cmff6\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.555790 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.555769    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:50.614252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.613862    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"]
Sep 11 18:01:50.617140 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:01:50.616337    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898d2f73_ea2b_4a8f_941a_1beb15768f7c.slice/crio-5f66ec596d194ddb3fe0a0e29d2f24da327514b64992cc7cdc3447d8ae028809 WatchSource:0}: Error finding container 5f66ec596d194ddb3fe0a0e29d2f24da327514b64992cc7cdc3447d8ae028809: Status 404 returned error can't find the container with id 5f66ec596d194ddb3fe0a0e29d2f24da327514b64992cc7cdc3447d8ae028809
Sep 11 18:01:50.669491 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.669468    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:50.707945 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.707911    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"]
Sep 11 18:01:50.709520 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:01:50.709488    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8eb91c3_06ad_4385_be14_d061616cca3e.slice/crio-9af58a565df4140bb97736b97c55af0ea70b0a061bab9946221a589d206f1224 WatchSource:0}: Error finding container 9af58a565df4140bb97736b97c55af0ea70b0a061bab9946221a589d206f1224: Status 404 returned error can't find the container with id 9af58a565df4140bb97736b97c55af0ea70b0a061bab9946221a589d206f1224
Sep 11 18:01:50.829995 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.829963    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"]
Sep 11 18:01:50.894200 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:01:50.894128    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9ba555_0b35_4a4e_8343_31d911a7d271.slice/crio-4e323d14618be1d8d03958c449dfe1dd364bcfd9fef7da8584f4d0f3d6615f1f WatchSource:0}: Error finding container 4e323d14618be1d8d03958c449dfe1dd364bcfd9fef7da8584f4d0f3d6615f1f: Status 404 returned error can't find the container with id 4e323d14618be1d8d03958c449dfe1dd364bcfd9fef7da8584f4d0f3d6615f1f
Sep 11 18:01:50.942339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:50.942292    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"]
Sep 11 18:01:50.943130 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:01:50.943104    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2fd5a4_2756_43c2_a8e7_6b7fa7b8bc57.slice/crio-4aa2d6c646caadce8b1418731df5f5e4db30ec776659fb5ea114f288dc6e9348 WatchSource:0}: Error finding container 4aa2d6c646caadce8b1418731df5f5e4db30ec776659fb5ea114f288dc6e9348: Status 404 returned error can't find the container with id 4aa2d6c646caadce8b1418731df5f5e4db30ec776659fb5ea114f288dc6e9348
Sep 11 18:01:51.569497 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.569461    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerID="4366c16f72e6de8117e614e022d074a9d19c74c1b379fe81fbaccc58f7c7b511" exitCode=0
Sep 11 18:01:51.569497 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.569491    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs" event={"ID":"8d9ba555-0b35-4a4e-8343-31d911a7d271","Type":"ContainerDied","Data":"4366c16f72e6de8117e614e022d074a9d19c74c1b379fe81fbaccc58f7c7b511"}
Sep 11 18:01:51.569949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.569540    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs" event={"ID":"8d9ba555-0b35-4a4e-8343-31d911a7d271","Type":"ContainerStarted","Data":"4e323d14618be1d8d03958c449dfe1dd364bcfd9fef7da8584f4d0f3d6615f1f"}
Sep 11 18:01:51.571167 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.571144    2571 generic.go:356] "Generic (PLEG): container finished" podID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerID="cacff8f57d92e70bb43fcb496866faff97fa779d8efd4008cc961d97d5b6d8d3" exitCode=0
Sep 11 18:01:51.571258 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.571224    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z" event={"ID":"898d2f73-ea2b-4a8f-941a-1beb15768f7c","Type":"ContainerDied","Data":"cacff8f57d92e70bb43fcb496866faff97fa779d8efd4008cc961d97d5b6d8d3"}
Sep 11 18:01:51.571337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.571271    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z" event={"ID":"898d2f73-ea2b-4a8f-941a-1beb15768f7c","Type":"ContainerStarted","Data":"5f66ec596d194ddb3fe0a0e29d2f24da327514b64992cc7cdc3447d8ae028809"}
Sep 11 18:01:51.572916 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.572892    2571 generic.go:356] "Generic (PLEG): container finished" podID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerID="28299284f45f952095947f1edf6d491deb8411a27cdc2d29bed6925415d60ae3" exitCode=0
Sep 11 18:01:51.572993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.572935    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6" event={"ID":"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57","Type":"ContainerDied","Data":"28299284f45f952095947f1edf6d491deb8411a27cdc2d29bed6925415d60ae3"}
Sep 11 18:01:51.572993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.572979    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6" event={"ID":"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57","Type":"ContainerStarted","Data":"4aa2d6c646caadce8b1418731df5f5e4db30ec776659fb5ea114f288dc6e9348"}
Sep 11 18:01:51.574616 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.574598    2571 generic.go:356] "Generic (PLEG): container finished" podID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerID="5eeed27f764d246218cb9cb16aedab75cfcefc22c5e419bc6e5497eb00c5f026" exitCode=0
Sep 11 18:01:51.574716 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.574638    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm" event={"ID":"c8eb91c3-06ad-4385-be14-d061616cca3e","Type":"ContainerDied","Data":"5eeed27f764d246218cb9cb16aedab75cfcefc22c5e419bc6e5497eb00c5f026"}
Sep 11 18:01:51.574716 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:51.574672    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm" event={"ID":"c8eb91c3-06ad-4385-be14-d061616cca3e","Type":"ContainerStarted","Data":"9af58a565df4140bb97736b97c55af0ea70b0a061bab9946221a589d206f1224"}
Sep 11 18:01:52.581463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:52.581369    2571 generic.go:356] "Generic (PLEG): container finished" podID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerID="f171d84818e58b6f6b051bd9922a86ef4c6716345ff926aab6c0988758146728" exitCode=0
Sep 11 18:01:52.581463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:52.581443    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z" event={"ID":"898d2f73-ea2b-4a8f-941a-1beb15768f7c","Type":"ContainerDied","Data":"f171d84818e58b6f6b051bd9922a86ef4c6716345ff926aab6c0988758146728"}
Sep 11 18:01:52.583332 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:52.583292    2571 generic.go:356] "Generic (PLEG): container finished" podID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerID="43e0f6c47eb24579828d58987c4826eb69e1f03039ffeec09e1552e411294576" exitCode=0
Sep 11 18:01:52.583407 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:52.583383    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6" event={"ID":"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57","Type":"ContainerDied","Data":"43e0f6c47eb24579828d58987c4826eb69e1f03039ffeec09e1552e411294576"}
Sep 11 18:01:53.589971 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:53.589935    2571 generic.go:356] "Generic (PLEG): container finished" podID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerID="0081792a7a6584ef2180d616f452e3af074e0c00e8986f04eec21832d71c56ad" exitCode=0
Sep 11 18:01:53.590401 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:53.590022    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6" event={"ID":"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57","Type":"ContainerDied","Data":"0081792a7a6584ef2180d616f452e3af074e0c00e8986f04eec21832d71c56ad"}
Sep 11 18:01:53.591788 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:53.591765    2571 generic.go:356] "Generic (PLEG): container finished" podID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerID="f117a7a533c37ee22e8a4925b85096a52ed4b2064a093376255c370eb296e3b9" exitCode=0
Sep 11 18:01:53.591868 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:53.591849    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm" event={"ID":"c8eb91c3-06ad-4385-be14-d061616cca3e","Type":"ContainerDied","Data":"f117a7a533c37ee22e8a4925b85096a52ed4b2064a093376255c370eb296e3b9"}
Sep 11 18:01:53.593885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:53.593866    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerID="b1028e78f2f8df39a9b18bda91d8cf73dda26f633aa35bcc8223044f107cc3c6" exitCode=0
Sep 11 18:01:53.594019 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:53.593995    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs" event={"ID":"8d9ba555-0b35-4a4e-8343-31d911a7d271","Type":"ContainerDied","Data":"b1028e78f2f8df39a9b18bda91d8cf73dda26f633aa35bcc8223044f107cc3c6"}
Sep 11 18:01:53.596315 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:53.596254    2571 generic.go:356] "Generic (PLEG): container finished" podID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerID="b559339d0911842ad808c1f17008863b86505d5324e4ffb9f067f793ea5ca80f" exitCode=0
Sep 11 18:01:53.596315 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:53.596287    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z" event={"ID":"898d2f73-ea2b-4a8f-941a-1beb15768f7c","Type":"ContainerDied","Data":"b559339d0911842ad808c1f17008863b86505d5324e4ffb9f067f793ea5ca80f"}
Sep 11 18:01:54.608596 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.608562    2571 generic.go:356] "Generic (PLEG): container finished" podID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerID="a25cf30f86c0528badf080882620dfbca4e47cf9d512da87ca6a25261ebbe407" exitCode=0
Sep 11 18:01:54.608984 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.608646    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs" event={"ID":"8d9ba555-0b35-4a4e-8343-31d911a7d271","Type":"ContainerDied","Data":"a25cf30f86c0528badf080882620dfbca4e47cf9d512da87ca6a25261ebbe407"}
Sep 11 18:01:54.610671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.610651    2571 generic.go:356] "Generic (PLEG): container finished" podID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerID="032bdf90a77b5458e3e81461e1f8f4268525c72984f82edfe232b59754dae518" exitCode=0
Sep 11 18:01:54.610751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.610727    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm" event={"ID":"c8eb91c3-06ad-4385-be14-d061616cca3e","Type":"ContainerDied","Data":"032bdf90a77b5458e3e81461e1f8f4268525c72984f82edfe232b59754dae518"}
Sep 11 18:01:54.738410 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.738383    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:54.756656 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.756635    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:54.766921 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.766897    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-util\") pod \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") "
Sep 11 18:01:54.767016 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.766961    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmff6\" (UniqueName: \"kubernetes.io/projected/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-kube-api-access-cmff6\") pod \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") "
Sep 11 18:01:54.767088 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.767026    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-bundle\") pod \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\" (UID: \"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57\") "
Sep 11 18:01:54.767667 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.767642    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-bundle" (OuterVolumeSpecName: "bundle") pod "2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" (UID: "2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:54.769060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.769038    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-kube-api-access-cmff6" (OuterVolumeSpecName: "kube-api-access-cmff6") pod "2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" (UID: "2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57"). InnerVolumeSpecName "kube-api-access-cmff6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:01:54.772542 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.772513    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-util" (OuterVolumeSpecName: "util") pod "2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" (UID: "2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:54.868224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.868155    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-bundle\") pod \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") "
Sep 11 18:01:54.868224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.868196    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-util\") pod \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") "
Sep 11 18:01:54.868408 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.868242    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97j5z\" (UniqueName: \"kubernetes.io/projected/898d2f73-ea2b-4a8f-941a-1beb15768f7c-kube-api-access-97j5z\") pod \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\" (UID: \"898d2f73-ea2b-4a8f-941a-1beb15768f7c\") "
Sep 11 18:01:54.868408 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.868394    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmff6\" (UniqueName: \"kubernetes.io/projected/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-kube-api-access-cmff6\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:54.868470 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.868411    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-bundle\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:54.868470 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.868422    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57-util\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:54.868662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.868628    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-bundle" (OuterVolumeSpecName: "bundle") pod "898d2f73-ea2b-4a8f-941a-1beb15768f7c" (UID: "898d2f73-ea2b-4a8f-941a-1beb15768f7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:54.870280 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.870257    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898d2f73-ea2b-4a8f-941a-1beb15768f7c-kube-api-access-97j5z" (OuterVolumeSpecName: "kube-api-access-97j5z") pod "898d2f73-ea2b-4a8f-941a-1beb15768f7c" (UID: "898d2f73-ea2b-4a8f-941a-1beb15768f7c"). InnerVolumeSpecName "kube-api-access-97j5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:01:54.873847 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.873828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-util" (OuterVolumeSpecName: "util") pod "898d2f73-ea2b-4a8f-941a-1beb15768f7c" (UID: "898d2f73-ea2b-4a8f-941a-1beb15768f7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:54.969488 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.969458    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-bundle\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:54.969488 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.969482    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/898d2f73-ea2b-4a8f-941a-1beb15768f7c-util\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:54.969488 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:54.969491    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97j5z\" (UniqueName: \"kubernetes.io/projected/898d2f73-ea2b-4a8f-941a-1beb15768f7c-kube-api-access-97j5z\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:55.617809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.617771    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z" event={"ID":"898d2f73-ea2b-4a8f-941a-1beb15768f7c","Type":"ContainerDied","Data":"5f66ec596d194ddb3fe0a0e29d2f24da327514b64992cc7cdc3447d8ae028809"}
Sep 11 18:01:55.618281 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.617820    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f66ec596d194ddb3fe0a0e29d2f24da327514b64992cc7cdc3447d8ae028809"
Sep 11 18:01:55.618281 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.617782    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xj67z"
Sep 11 18:01:55.619814 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.619797    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6"
Sep 11 18:01:55.619905 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.619843    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acjkn6" event={"ID":"2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57","Type":"ContainerDied","Data":"4aa2d6c646caadce8b1418731df5f5e4db30ec776659fb5ea114f288dc6e9348"}
Sep 11 18:01:55.619905 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.619883    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4aa2d6c646caadce8b1418731df5f5e4db30ec776659fb5ea114f288dc6e9348"
Sep 11 18:01:55.763900 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.763878    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:55.765921 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.765905    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:55.876212 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.876133    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4zs7\" (UniqueName: \"kubernetes.io/projected/c8eb91c3-06ad-4385-be14-d061616cca3e-kube-api-access-r4zs7\") pod \"c8eb91c3-06ad-4385-be14-d061616cca3e\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") "
Sep 11 18:01:55.876212 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.876169    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-bundle\") pod \"8d9ba555-0b35-4a4e-8343-31d911a7d271\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") "
Sep 11 18:01:55.876454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.876216    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-util\") pod \"c8eb91c3-06ad-4385-be14-d061616cca3e\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") "
Sep 11 18:01:55.876454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.876242    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-bundle\") pod \"c8eb91c3-06ad-4385-be14-d061616cca3e\" (UID: \"c8eb91c3-06ad-4385-be14-d061616cca3e\") "
Sep 11 18:01:55.876454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.876264    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-util\") pod \"8d9ba555-0b35-4a4e-8343-31d911a7d271\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") "
Sep 11 18:01:55.876454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.876317    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz7hj\" (UniqueName: \"kubernetes.io/projected/8d9ba555-0b35-4a4e-8343-31d911a7d271-kube-api-access-lz7hj\") pod \"8d9ba555-0b35-4a4e-8343-31d911a7d271\" (UID: \"8d9ba555-0b35-4a4e-8343-31d911a7d271\") "
Sep 11 18:01:55.876793 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.876765    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-bundle" (OuterVolumeSpecName: "bundle") pod "c8eb91c3-06ad-4385-be14-d061616cca3e" (UID: "c8eb91c3-06ad-4385-be14-d061616cca3e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:55.876884 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.876852    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-bundle" (OuterVolumeSpecName: "bundle") pod "8d9ba555-0b35-4a4e-8343-31d911a7d271" (UID: "8d9ba555-0b35-4a4e-8343-31d911a7d271"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:55.878333 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.878293    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8eb91c3-06ad-4385-be14-d061616cca3e-kube-api-access-r4zs7" (OuterVolumeSpecName: "kube-api-access-r4zs7") pod "c8eb91c3-06ad-4385-be14-d061616cca3e" (UID: "c8eb91c3-06ad-4385-be14-d061616cca3e"). InnerVolumeSpecName "kube-api-access-r4zs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:01:55.878822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.878805    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9ba555-0b35-4a4e-8343-31d911a7d271-kube-api-access-lz7hj" (OuterVolumeSpecName: "kube-api-access-lz7hj") pod "8d9ba555-0b35-4a4e-8343-31d911a7d271" (UID: "8d9ba555-0b35-4a4e-8343-31d911a7d271"). InnerVolumeSpecName "kube-api-access-lz7hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:01:55.882343 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.882227    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-util" (OuterVolumeSpecName: "util") pod "8d9ba555-0b35-4a4e-8343-31d911a7d271" (UID: "8d9ba555-0b35-4a4e-8343-31d911a7d271"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:55.882653 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.882631    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-util" (OuterVolumeSpecName: "util") pod "c8eb91c3-06ad-4385-be14-d061616cca3e" (UID: "c8eb91c3-06ad-4385-be14-d061616cca3e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:01:55.977365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.977333    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-util\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:55.977365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.977359    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8eb91c3-06ad-4385-be14-d061616cca3e-bundle\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:55.977365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.977368    2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-util\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:55.977578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.977378    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz7hj\" (UniqueName: \"kubernetes.io/projected/8d9ba555-0b35-4a4e-8343-31d911a7d271-kube-api-access-lz7hj\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:55.977578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.977387    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4zs7\" (UniqueName: \"kubernetes.io/projected/c8eb91c3-06ad-4385-be14-d061616cca3e-kube-api-access-r4zs7\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:55.977578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:55.977405    2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8d9ba555-0b35-4a4e-8343-31d911a7d271-bundle\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:01:56.626864 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:56.626834    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm"
Sep 11 18:01:56.627347 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:56.626861    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x55qm" event={"ID":"c8eb91c3-06ad-4385-be14-d061616cca3e","Type":"ContainerDied","Data":"9af58a565df4140bb97736b97c55af0ea70b0a061bab9946221a589d206f1224"}
Sep 11 18:01:56.627347 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:56.626909    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af58a565df4140bb97736b97c55af0ea70b0a061bab9946221a589d206f1224"
Sep 11 18:01:56.628715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:56.628695    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs" event={"ID":"8d9ba555-0b35-4a4e-8343-31d911a7d271","Type":"ContainerDied","Data":"4e323d14618be1d8d03958c449dfe1dd364bcfd9fef7da8584f4d0f3d6615f1f"}
Sep 11 18:01:56.628837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:56.628720    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cj99xs"
Sep 11 18:01:56.628837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:01:56.628726    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e323d14618be1d8d03958c449dfe1dd364bcfd9fef7da8584f4d0f3d6615f1f"
Sep 11 18:02:04.803340 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803308    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"]
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803599    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerName="util"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803612    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerName="util"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803622    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerName="extract"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803627    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerName="extract"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803636    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerName="util"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803641    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerName="util"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803651    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerName="pull"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803656    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerName="pull"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803663    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerName="pull"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803669    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerName="pull"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803677    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerName="pull"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803686    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerName="pull"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803694    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerName="extract"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803700    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerName="extract"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803708    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerName="extract"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803712    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerName="extract"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803717    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerName="extract"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803722    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerName="extract"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803733    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerName="util"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803738    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerName="util"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803744    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerName="pull"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803749    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerName="pull"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803762    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerName="util"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803769    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerName="util"
Sep 11 18:02:04.803800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803811    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="898d2f73-ea2b-4a8f-941a-1beb15768f7c" containerName="extract"
Sep 11 18:02:04.804545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803818    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d9ba555-0b35-4a4e-8343-31d911a7d271" containerName="extract"
Sep 11 18:02:04.804545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803825    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8eb91c3-06ad-4385-be14-d061616cca3e" containerName="extract"
Sep 11 18:02:04.804545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.803830    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a2fd5a4-2756-43c2-a8e7-6b7fa7b8bc57" containerName="extract"
Sep 11 18:02:04.807925 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.807908    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:04.810662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.810636    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:02:04.810777 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.810742    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jpgrj\""
Sep 11 18:02:04.810777 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.810756    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:02:04.820315 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.820273    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"]
Sep 11 18:02:04.847651 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.847630    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbmv\" (UniqueName: \"kubernetes.io/projected/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-kube-api-access-6vbmv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" (UID: \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:04.847758 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.847684    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" (UID: \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:04.948810 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.948770    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbmv\" (UniqueName: \"kubernetes.io/projected/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-kube-api-access-6vbmv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" (UID: \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:04.949014 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.948835    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" (UID: \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:04.949238 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.949218    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" (UID: \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:04.959995 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:04.959971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbmv\" (UniqueName: \"kubernetes.io/projected/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-kube-api-access-6vbmv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" (UID: \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:05.117466 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:05.117390    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:05.394180 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:05.394127    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"]
Sep 11 18:02:05.396402 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:05.396375    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0b6915_99e5_428f_bd69_ac2cb45cbbe9.slice/crio-4b61a85d938e7d50b64f28415fb1b567fcca9f2c1558fa35838803b476ca9c63 WatchSource:0}: Error finding container 4b61a85d938e7d50b64f28415fb1b567fcca9f2c1558fa35838803b476ca9c63: Status 404 returned error can't find the container with id 4b61a85d938e7d50b64f28415fb1b567fcca9f2c1558fa35838803b476ca9c63
Sep 11 18:02:05.671795 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:05.671764    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg" event={"ID":"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9","Type":"ContainerStarted","Data":"4b61a85d938e7d50b64f28415fb1b567fcca9f2c1558fa35838803b476ca9c63"}
Sep 11 18:02:06.531576 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.531540    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"]
Sep 11 18:02:06.535097 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.535009    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"
Sep 11 18:02:06.538137 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.537948    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 18:02:06.538137 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.538101    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rmjv8\""
Sep 11 18:02:06.547600 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.547541    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"]
Sep 11 18:02:06.663023 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.662988    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4fr\" (UniqueName: \"kubernetes.io/projected/0bd0932b-c6d5-4a02-837e-a96aca56a175-kube-api-access-ks4fr\") pod \"dns-operator-controller-manager-7b84cd7c9c-k5xlt\" (UID: \"0bd0932b-c6d5-4a02-837e-a96aca56a175\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"
Sep 11 18:02:06.764952 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.764901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4fr\" (UniqueName: \"kubernetes.io/projected/0bd0932b-c6d5-4a02-837e-a96aca56a175-kube-api-access-ks4fr\") pod \"dns-operator-controller-manager-7b84cd7c9c-k5xlt\" (UID: \"0bd0932b-c6d5-4a02-837e-a96aca56a175\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"
Sep 11 18:02:06.787634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.787477    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4fr\" (UniqueName: \"kubernetes.io/projected/0bd0932b-c6d5-4a02-837e-a96aca56a175-kube-api-access-ks4fr\") pod \"dns-operator-controller-manager-7b84cd7c9c-k5xlt\" (UID: \"0bd0932b-c6d5-4a02-837e-a96aca56a175\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"
Sep 11 18:02:06.849040 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:06.849000    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"
Sep 11 18:02:07.133388 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:07.133355    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"]
Sep 11 18:02:07.135692 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:07.135662    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd0932b_c6d5_4a02_837e_a96aca56a175.slice/crio-4a0a865ad3aec7959f3b86ef6669767115795467c800d27c58e0da9a693ff9fe WatchSource:0}: Error finding container 4a0a865ad3aec7959f3b86ef6669767115795467c800d27c58e0da9a693ff9fe: Status 404 returned error can't find the container with id 4a0a865ad3aec7959f3b86ef6669767115795467c800d27c58e0da9a693ff9fe
Sep 11 18:02:07.689283 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:07.689236    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt" event={"ID":"0bd0932b-c6d5-4a02-837e-a96aca56a175","Type":"ContainerStarted","Data":"4a0a865ad3aec7959f3b86ef6669767115795467c800d27c58e0da9a693ff9fe"}
Sep 11 18:02:09.318456 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:09.318417    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f8bxx"]
Sep 11 18:02:09.321923 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:09.321902    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx"
Sep 11 18:02:09.324634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:09.324603    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jhhrw\""
Sep 11 18:02:09.331765 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:09.331745    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f8bxx"]
Sep 11 18:02:09.387808 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:09.387779    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6jx\" (UniqueName: \"kubernetes.io/projected/3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e-kube-api-access-dv6jx\") pod \"authorino-operator-76b478bd78-f8bxx\" (UID: \"3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e\") " pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx"
Sep 11 18:02:09.488931 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:09.488888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6jx\" (UniqueName: \"kubernetes.io/projected/3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e-kube-api-access-dv6jx\") pod \"authorino-operator-76b478bd78-f8bxx\" (UID: \"3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e\") " pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx"
Sep 11 18:02:09.497520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:09.497496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6jx\" (UniqueName: \"kubernetes.io/projected/3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e-kube-api-access-dv6jx\") pod \"authorino-operator-76b478bd78-f8bxx\" (UID: \"3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e\") " pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx"
Sep 11 18:02:09.635316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:09.635236    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx"
Sep 11 18:02:10.331135 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:10.331112    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f8bxx"]
Sep 11 18:02:10.333022 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:10.332995    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca65cf0_5e32_44d9_ac74_fb0e7f97c46e.slice/crio-90511fd32876e61897da115e739c064512ed1c06c0a036ded196e87d2262edd1 WatchSource:0}: Error finding container 90511fd32876e61897da115e739c064512ed1c06c0a036ded196e87d2262edd1: Status 404 returned error can't find the container with id 90511fd32876e61897da115e739c064512ed1c06c0a036ded196e87d2262edd1
Sep 11 18:02:10.708162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:10.708122    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx" event={"ID":"3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e","Type":"ContainerStarted","Data":"90511fd32876e61897da115e739c064512ed1c06c0a036ded196e87d2262edd1"}
Sep 11 18:02:10.709774 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:10.709743    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg" event={"ID":"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9","Type":"ContainerStarted","Data":"bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf"}
Sep 11 18:02:10.709962 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:10.709800    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:10.711221 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:10.711197    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt" event={"ID":"0bd0932b-c6d5-4a02-837e-a96aca56a175","Type":"ContainerStarted","Data":"6aa5a3e26d3f98544c5c17a58785bb2af23472ce28515ddd28d1c38be6d6c6be"}
Sep 11 18:02:10.711333 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:10.711325    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"
Sep 11 18:02:10.755546 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:10.755498    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg" podStartSLOduration=2.08243507 podStartE2EDuration="6.755482296s" totalImagesPullingTime="4.673047226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:10.754473066 +0000 UTC m=+494.524539436" watchObservedRunningTime="2026-09-11 18:02:10.755482296 +0000 UTC m=+494.525548644"
Sep 11 18:02:10.783515 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:10.783466    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt" podStartSLOduration=1.8536957950000001 podStartE2EDuration="4.783449738s" totalImagesPullingTime="2.929753943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:10.781489185 +0000 UTC m=+494.551555545" watchObservedRunningTime="2026-09-11 18:02:10.783449738 +0000 UTC m=+494.553516085"
Sep 11 18:02:11.502498 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.500341    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"]
Sep 11 18:02:11.505055 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.505025    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:11.508059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.508040    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jszbp\""
Sep 11 18:02:11.514330 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.514309    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"]
Sep 11 18:02:11.608100 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.608077    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q79k\" (UniqueName: \"kubernetes.io/projected/b8f68656-5be9-471b-8039-647fdcde9a6f-kube-api-access-6q79k\") pod \"limitador-operator-controller-manager-74495c69f8-58p5v\" (UID: \"b8f68656-5be9-471b-8039-647fdcde9a6f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:11.708769 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.708731    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q79k\" (UniqueName: \"kubernetes.io/projected/b8f68656-5be9-471b-8039-647fdcde9a6f-kube-api-access-6q79k\") pod \"limitador-operator-controller-manager-74495c69f8-58p5v\" (UID: \"b8f68656-5be9-471b-8039-647fdcde9a6f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:11.717577 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.717545    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx" event={"ID":"3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e","Type":"ContainerStarted","Data":"9cdbb577529b96be246f1f9e9de23e9128ac8b01ce1d0bfbdad4d9184e30ee98"}
Sep 11 18:02:11.717955 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.717929    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx"
Sep 11 18:02:11.719569 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.719544    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q79k\" (UniqueName: \"kubernetes.io/projected/b8f68656-5be9-471b-8039-647fdcde9a6f-kube-api-access-6q79k\") pod \"limitador-operator-controller-manager-74495c69f8-58p5v\" (UID: \"b8f68656-5be9-471b-8039-647fdcde9a6f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:11.735534 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.735496    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx" podStartSLOduration=1.52452004 podStartE2EDuration="2.735482219s" totalImagesPullingTime="1.210962179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:11.734991715 +0000 UTC m=+495.505058077" watchObservedRunningTime="2026-09-11 18:02:11.735482219 +0000 UTC m=+495.505548567"
Sep 11 18:02:11.815712 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:11.815649    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:12.091152 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:12.091127    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"]
Sep 11 18:02:12.092879 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:12.092844    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f68656_5be9_471b_8039_647fdcde9a6f.slice/crio-5459bc4ff33283518919bd9d8de241d8eee13ca18d01b40de14f982b52267701 WatchSource:0}: Error finding container 5459bc4ff33283518919bd9d8de241d8eee13ca18d01b40de14f982b52267701: Status 404 returned error can't find the container with id 5459bc4ff33283518919bd9d8de241d8eee13ca18d01b40de14f982b52267701
Sep 11 18:02:12.732141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:12.732110    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v" event={"ID":"b8f68656-5be9-471b-8039-647fdcde9a6f","Type":"ContainerStarted","Data":"5459bc4ff33283518919bd9d8de241d8eee13ca18d01b40de14f982b52267701"}
Sep 11 18:02:14.735103 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:14.735018    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v" event={"ID":"b8f68656-5be9-471b-8039-647fdcde9a6f","Type":"ContainerStarted","Data":"2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a"}
Sep 11 18:02:14.735463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:14.735115    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:14.754682 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:14.754640    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v" podStartSLOduration=1.365843022 podStartE2EDuration="3.754626764s" totalImagesPullingTime="2.388783742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:14.753124949 +0000 UTC m=+498.523191297" watchObservedRunningTime="2026-09-11 18:02:14.754626764 +0000 UTC m=+498.524693113"
Sep 11 18:02:21.719895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:21.719866    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:21.720319 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:21.720041    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-k5xlt"
Sep 11 18:02:23.731774 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.731735    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-f8bxx"
Sep 11 18:02:23.735290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.735265    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"]
Sep 11 18:02:23.738755 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.738739    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"
Sep 11 18:02:23.750068 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.750044    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"]
Sep 11 18:02:23.783885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.783856    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"]
Sep 11 18:02:23.784155 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:23.784132    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-2xs45], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-2xs45]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj" podUID="1a8868ff-c81a-4ff8-8d73-10e01f181204"
Sep 11 18:02:23.795145 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.795119    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"]
Sep 11 18:02:23.814261 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.814236    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"]
Sep 11 18:02:23.814485 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.814463    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" containerName="manager" containerID="cri-o://bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf" gracePeriod=2
Sep 11 18:02:23.838941 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.838914    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"]
Sep 11 18:02:23.839762 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:23.839731    2571 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-hrphg\": pods \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:23.841230 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.841201    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"]
Sep 11 18:02:23.841636 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.841619    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" containerName="manager"
Sep 11 18:02:23.841636 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.841638    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" containerName="manager"
Sep 11 18:02:23.841725 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.841705    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" containerName="manager"
Sep 11 18:02:23.844523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.844510    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:23.847135 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:23.847094    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:23.847737 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.847718    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"]
Sep 11 18:02:23.850772 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.850755    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"]
Sep 11 18:02:23.850861 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.850842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:23.850921 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.850903    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" containerName="manager" containerID="cri-o://2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a" gracePeriod=2
Sep 11 18:02:23.853074 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.853052    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:23.855125 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.855106    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"]
Sep 11 18:02:23.860914 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.860442    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"]
Sep 11 18:02:23.865281 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.865261    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"]
Sep 11 18:02:23.872243 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:23.872212    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-58p5v\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:23.874212 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:23.874186    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:23.879145 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.879125    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"]
Sep 11 18:02:23.879543 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.879518    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" containerName="manager"
Sep 11 18:02:23.879543 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.879544    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" containerName="manager"
Sep 11 18:02:23.879649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.879618    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" containerName="manager"
Sep 11 18:02:23.882559 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.882543    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"
Sep 11 18:02:23.892887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:23.892867    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"]
Sep 11 18:02:23.895977 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:23.895947    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:23.928785 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:23.928751    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-58p5v\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:24.005026 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.004949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpbl\" (UniqueName: \"kubernetes.io/projected/2469f45e-5948-4024-8c59-a57a7ea18be2-kube-api-access-qxpbl\") pod \"limitador-operator-controller-manager-74495c69f8-dl7xx\" (UID: \"2469f45e-5948-4024-8c59-a57a7ea18be2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"
Sep 11 18:02:24.005026 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.004988    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdw7z\" (UniqueName: \"kubernetes.io/projected/64419844-ea38-4dc6-b2bd-6b5e66eaa060-kube-api-access-gdw7z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbfc\" (UID: \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:24.005207 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.005027    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxzk5\" (UniqueName: \"kubernetes.io/projected/7b0592ba-4253-4896-bf50-b427cbe39bc1-kube-api-access-qxzk5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dlrfx\" (UID: \"7b0592ba-4253-4896-bf50-b427cbe39bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:24.005207 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.005082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b0592ba-4253-4896-bf50-b427cbe39bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dlrfx\" (UID: \"7b0592ba-4253-4896-bf50-b427cbe39bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:24.005207 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.005152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64419844-ea38-4dc6-b2bd-6b5e66eaa060-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbfc\" (UID: \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:24.090449 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.090426    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:24.093147 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:24.093117    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-58p5v\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:24.095231 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:24.095206    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:24.106478 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.106457    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdw7z\" (UniqueName: \"kubernetes.io/projected/64419844-ea38-4dc6-b2bd-6b5e66eaa060-kube-api-access-gdw7z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbfc\" (UID: \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:24.106589 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.106496    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxzk5\" (UniqueName: \"kubernetes.io/projected/7b0592ba-4253-4896-bf50-b427cbe39bc1-kube-api-access-qxzk5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dlrfx\" (UID: \"7b0592ba-4253-4896-bf50-b427cbe39bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:24.106589 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.106541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b0592ba-4253-4896-bf50-b427cbe39bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dlrfx\" (UID: \"7b0592ba-4253-4896-bf50-b427cbe39bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:24.106589 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.106568    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64419844-ea38-4dc6-b2bd-6b5e66eaa060-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbfc\" (UID: \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:24.106763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.106597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpbl\" (UniqueName: \"kubernetes.io/projected/2469f45e-5948-4024-8c59-a57a7ea18be2-kube-api-access-qxpbl\") pod \"limitador-operator-controller-manager-74495c69f8-dl7xx\" (UID: \"2469f45e-5948-4024-8c59-a57a7ea18be2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"
Sep 11 18:02:24.106951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.106934    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64419844-ea38-4dc6-b2bd-6b5e66eaa060-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbfc\" (UID: \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:24.106951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.106942    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b0592ba-4253-4896-bf50-b427cbe39bc1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dlrfx\" (UID: \"7b0592ba-4253-4896-bf50-b427cbe39bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:24.111639 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.111622    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:24.113929 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:24.113906    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-58p5v\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:24.115897 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:24.115872    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-hrphg\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:24.116878 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.116862    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdw7z\" (UniqueName: \"kubernetes.io/projected/64419844-ea38-4dc6-b2bd-6b5e66eaa060-kube-api-access-gdw7z\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rhbfc\" (UID: \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:24.117112 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.117095    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxzk5\" (UniqueName: \"kubernetes.io/projected/7b0592ba-4253-4896-bf50-b427cbe39bc1-kube-api-access-qxzk5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dlrfx\" (UID: \"7b0592ba-4253-4896-bf50-b427cbe39bc1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:24.117162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.117097    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpbl\" (UniqueName: \"kubernetes.io/projected/2469f45e-5948-4024-8c59-a57a7ea18be2-kube-api-access-qxpbl\") pod \"limitador-operator-controller-manager-74495c69f8-dl7xx\" (UID: \"2469f45e-5948-4024-8c59-a57a7ea18be2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"
Sep 11 18:02:24.207837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.207811    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vbmv\" (UniqueName: \"kubernetes.io/projected/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-kube-api-access-6vbmv\") pod \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\" (UID: \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\") "
Sep 11 18:02:24.207991 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.207856    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-extensions-socket-volume\") pod \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\" (UID: \"ea0b6915-99e5-428f-bd69-ac2cb45cbbe9\") "
Sep 11 18:02:24.207991 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.207892    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q79k\" (UniqueName: \"kubernetes.io/projected/b8f68656-5be9-471b-8039-647fdcde9a6f-kube-api-access-6q79k\") pod \"b8f68656-5be9-471b-8039-647fdcde9a6f\" (UID: \"b8f68656-5be9-471b-8039-647fdcde9a6f\") "
Sep 11 18:02:24.208417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.208392    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" (UID: "ea0b6915-99e5-428f-bd69-ac2cb45cbbe9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:02:24.209915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.209892    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-kube-api-access-6vbmv" (OuterVolumeSpecName: "kube-api-access-6vbmv") pod "ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" (UID: "ea0b6915-99e5-428f-bd69-ac2cb45cbbe9"). InnerVolumeSpecName "kube-api-access-6vbmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:02:24.209971 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.209925    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f68656-5be9-471b-8039-647fdcde9a6f-kube-api-access-6q79k" (OuterVolumeSpecName: "kube-api-access-6q79k") pod "b8f68656-5be9-471b-8039-647fdcde9a6f" (UID: "b8f68656-5be9-471b-8039-647fdcde9a6f"). InnerVolumeSpecName "kube-api-access-6q79k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:02:24.258280 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.258215    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:24.267172 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.267148    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:24.272183 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.272163    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"
Sep 11 18:02:24.309282 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.309246    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vbmv\" (UniqueName: \"kubernetes.io/projected/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-kube-api-access-6vbmv\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:02:24.309486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.309317    2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9-extensions-socket-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:02:24.309486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.309335    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q79k\" (UniqueName: \"kubernetes.io/projected/b8f68656-5be9-471b-8039-647fdcde9a6f-kube-api-access-6q79k\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:02:24.544477 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:24.544448    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64419844_ea38_4dc6_b2bd_6b5e66eaa060.slice/crio-d67627ebec798df384f2784dc12875f23dd9e789099195f746ecc2cb50061b0d WatchSource:0}: Error finding container d67627ebec798df384f2784dc12875f23dd9e789099195f746ecc2cb50061b0d: Status 404 returned error can't find the container with id d67627ebec798df384f2784dc12875f23dd9e789099195f746ecc2cb50061b0d
Sep 11 18:02:24.545169 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.545145    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"]
Sep 11 18:02:24.549860 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.549837    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"]
Sep 11 18:02:24.550630 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:24.550609    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0592ba_4253_4896_bf50_b427cbe39bc1.slice/crio-e807925110ab5682f7e89f73ca0d250ed34e851b2ad305df2dc1fab6ebde86b3 WatchSource:0}: Error finding container e807925110ab5682f7e89f73ca0d250ed34e851b2ad305df2dc1fab6ebde86b3: Status 404 returned error can't find the container with id e807925110ab5682f7e89f73ca0d250ed34e851b2ad305df2dc1fab6ebde86b3
Sep 11 18:02:24.565743 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.565720    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"]
Sep 11 18:02:24.569443 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:24.569423    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2469f45e_5948_4024_8c59_a57a7ea18be2.slice/crio-4e98a7be0e4c618f4da9f0d8c0abffc8c2390215a199fa053686c025e9f099fc WatchSource:0}: Error finding container 4e98a7be0e4c618f4da9f0d8c0abffc8c2390215a199fa053686c025e9f099fc: Status 404 returned error can't find the container with id 4e98a7be0e4c618f4da9f0d8c0abffc8c2390215a199fa053686c025e9f099fc
Sep 11 18:02:24.730723 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.730683    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8868ff-c81a-4ff8-8d73-10e01f181204" path="/var/lib/kubelet/pods/1a8868ff-c81a-4ff8-8d73-10e01f181204/volumes"
Sep 11 18:02:24.730885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.730873    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f68656-5be9-471b-8039-647fdcde9a6f" path="/var/lib/kubelet/pods/b8f68656-5be9-471b-8039-647fdcde9a6f/volumes"
Sep 11 18:02:24.731231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.731209    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" path="/var/lib/kubelet/pods/ea0b6915-99e5-428f-bd69-ac2cb45cbbe9/volumes"
Sep 11 18:02:24.792134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.792058    2571 generic.go:356] "Generic (PLEG): container finished" podID="ea0b6915-99e5-428f-bd69-ac2cb45cbbe9" containerID="bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf" exitCode=0
Sep 11 18:02:24.792134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.792126    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hrphg"
Sep 11 18:02:24.792648 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.792141    2571 scope.go:117] "RemoveContainer" containerID="bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf"
Sep 11 18:02:24.794230 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.794201    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc" event={"ID":"64419844-ea38-4dc6-b2bd-6b5e66eaa060","Type":"ContainerStarted","Data":"448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98"}
Sep 11 18:02:24.794365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.794262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc" event={"ID":"64419844-ea38-4dc6-b2bd-6b5e66eaa060","Type":"ContainerStarted","Data":"d67627ebec798df384f2784dc12875f23dd9e789099195f746ecc2cb50061b0d"}
Sep 11 18:02:24.794365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.794289    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:24.796127 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.796103    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx" event={"ID":"2469f45e-5948-4024-8c59-a57a7ea18be2","Type":"ContainerStarted","Data":"85433aac32253d8b28eb1ab24b64fe1e6bdebf22ff75ba9ea8d232e288e871c2"}
Sep 11 18:02:24.796234 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.796141    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx" event={"ID":"2469f45e-5948-4024-8c59-a57a7ea18be2","Type":"ContainerStarted","Data":"4e98a7be0e4c618f4da9f0d8c0abffc8c2390215a199fa053686c025e9f099fc"}
Sep 11 18:02:24.796234 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.796168    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"
Sep 11 18:02:24.797961 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.797938    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx" event={"ID":"7b0592ba-4253-4896-bf50-b427cbe39bc1","Type":"ContainerStarted","Data":"f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc"}
Sep 11 18:02:24.798066 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.797983    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx" event={"ID":"7b0592ba-4253-4896-bf50-b427cbe39bc1","Type":"ContainerStarted","Data":"e807925110ab5682f7e89f73ca0d250ed34e851b2ad305df2dc1fab6ebde86b3"}
Sep 11 18:02:24.798066 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.798014    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:24.799218 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.799200    2571 generic.go:356] "Generic (PLEG): container finished" podID="b8f68656-5be9-471b-8039-647fdcde9a6f" containerID="2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a" exitCode=0
Sep 11 18:02:24.799317 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.799242    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-58p5v"
Sep 11 18:02:24.799360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.799321    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"
Sep 11 18:02:24.803891 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.803870    2571 scope.go:117] "RemoveContainer" containerID="bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf"
Sep 11 18:02:24.804428 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:24.804402    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf\": container with ID starting with bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf not found: ID does not exist" containerID="bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf"
Sep 11 18:02:24.804530 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.804437    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf"} err="failed to get container status \"bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf\": rpc error: code = NotFound desc = could not find container \"bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf\": container with ID starting with bc6f74e3c1caf4e89ff6abd6f4a24590476fe091df6d624521bc524bc2ffcfbf not found: ID does not exist"
Sep 11 18:02:24.804530 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.804463    2571 scope.go:117] "RemoveContainer" containerID="2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a"
Sep 11 18:02:24.811595 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.811576    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"
Sep 11 18:02:24.813439 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.813418    2571 scope.go:117] "RemoveContainer" containerID="2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a"
Sep 11 18:02:24.813790 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:24.813773    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a\": container with ID starting with 2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a not found: ID does not exist" containerID="2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a"
Sep 11 18:02:24.813835 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.813796    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a"} err="failed to get container status \"2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a\": rpc error: code = NotFound desc = could not find container \"2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a\": container with ID starting with 2e480f583346a138a5f879375b6f8eb6f9dd39d27cad0a058d0e2f333a2a3e7a not found: ID does not exist"
Sep 11 18:02:24.853337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.853272    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc" podStartSLOduration=1.853250821 podStartE2EDuration="1.853250821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:24.850952295 +0000 UTC m=+508.621018645" watchObservedRunningTime="2026-09-11 18:02:24.853250821 +0000 UTC m=+508.623317171"
Sep 11 18:02:24.889122 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.889052    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx" podStartSLOduration=1.8890306159999999 podStartE2EDuration="1.889030616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:24.884560048 +0000 UTC m=+508.654626397" watchObservedRunningTime="2026-09-11 18:02:24.889030616 +0000 UTC m=+508.659096966"
Sep 11 18:02:24.932104 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:24.932065    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-shztj\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="1a8868ff-c81a-4ff8-8d73-10e01f181204" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"
Sep 11 18:02:24.936374 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:24.936331    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx" podStartSLOduration=1.936316484 podStartE2EDuration="1.936316484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:24.928872206 +0000 UTC m=+508.698938555" watchObservedRunningTime="2026-09-11 18:02:24.936316484 +0000 UTC m=+508.706382823"
Sep 11 18:02:25.808590 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:25.808564    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"
Sep 11 18:02:25.811113 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:25.811073    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-shztj\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="1a8868ff-c81a-4ff8-8d73-10e01f181204" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"
Sep 11 18:02:25.816736 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:25.816706    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-shztj\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="1a8868ff-c81a-4ff8-8d73-10e01f181204" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"
Sep 11 18:02:26.730849 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:26.730810    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-shztj\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="1a8868ff-c81a-4ff8-8d73-10e01f181204" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-shztj"
Sep 11 18:02:35.811238 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:35.811155    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dl7xx"
Sep 11 18:02:35.811695 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:35.811334    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:02:35.811695 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:35.811385    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:35.886714 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:35.886681    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"]
Sep 11 18:02:35.886948 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:35.886918    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx" podUID="7b0592ba-4253-4896-bf50-b427cbe39bc1" containerName="manager" containerID="cri-o://f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc" gracePeriod=10
Sep 11 18:02:36.126389 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.126363    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:36.307581 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.307548    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxzk5\" (UniqueName: \"kubernetes.io/projected/7b0592ba-4253-4896-bf50-b427cbe39bc1-kube-api-access-qxzk5\") pod \"7b0592ba-4253-4896-bf50-b427cbe39bc1\" (UID: \"7b0592ba-4253-4896-bf50-b427cbe39bc1\") "
Sep 11 18:02:36.307749 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.307615    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b0592ba-4253-4896-bf50-b427cbe39bc1-extensions-socket-volume\") pod \"7b0592ba-4253-4896-bf50-b427cbe39bc1\" (UID: \"7b0592ba-4253-4896-bf50-b427cbe39bc1\") "
Sep 11 18:02:36.307966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.307943    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b0592ba-4253-4896-bf50-b427cbe39bc1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7b0592ba-4253-4896-bf50-b427cbe39bc1" (UID: "7b0592ba-4253-4896-bf50-b427cbe39bc1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:02:36.309612 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.309585    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0592ba-4253-4896-bf50-b427cbe39bc1-kube-api-access-qxzk5" (OuterVolumeSpecName: "kube-api-access-qxzk5") pod "7b0592ba-4253-4896-bf50-b427cbe39bc1" (UID: "7b0592ba-4253-4896-bf50-b427cbe39bc1"). InnerVolumeSpecName "kube-api-access-qxzk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:02:36.408376 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.408344    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxzk5\" (UniqueName: \"kubernetes.io/projected/7b0592ba-4253-4896-bf50-b427cbe39bc1-kube-api-access-qxzk5\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:02:36.408376 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.408375    2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b0592ba-4253-4896-bf50-b427cbe39bc1-extensions-socket-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:02:36.868502 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.868464    2571 generic.go:356] "Generic (PLEG): container finished" podID="7b0592ba-4253-4896-bf50-b427cbe39bc1" containerID="f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc" exitCode=0
Sep 11 18:02:36.868933 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.868508    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"
Sep 11 18:02:36.868933 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.868512    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx" event={"ID":"7b0592ba-4253-4896-bf50-b427cbe39bc1","Type":"ContainerDied","Data":"f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc"}
Sep 11 18:02:36.868933 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.868563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx" event={"ID":"7b0592ba-4253-4896-bf50-b427cbe39bc1","Type":"ContainerDied","Data":"e807925110ab5682f7e89f73ca0d250ed34e851b2ad305df2dc1fab6ebde86b3"}
Sep 11 18:02:36.868933 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.868586    2571 scope.go:117] "RemoveContainer" containerID="f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc"
Sep 11 18:02:36.877333 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.877312    2571 scope.go:117] "RemoveContainer" containerID="f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc"
Sep 11 18:02:36.877633 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:02:36.877602    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc\": container with ID starting with f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc not found: ID does not exist" containerID="f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc"
Sep 11 18:02:36.877732 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.877636    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc"} err="failed to get container status \"f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc\": rpc error: code = NotFound desc = could not find container \"f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc\": container with ID starting with f36b2f0950bff275731ee49cf2ca9ddc3c5238a767664516ceb25b86f83a08fc not found: ID does not exist"
Sep 11 18:02:36.891176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.891149    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"]
Sep 11 18:02:36.900089 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:36.900068    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dlrfx"]
Sep 11 18:02:38.729868 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:38.729835    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0592ba-4253-4896-bf50-b427cbe39bc1" path="/var/lib/kubelet/pods/7b0592ba-4253-4896-bf50-b427cbe39bc1/volumes"
Sep 11 18:02:52.216629 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.216545    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"]
Sep 11 18:02:52.217060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.217041    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0592ba-4253-4896-bf50-b427cbe39bc1" containerName="manager"
Sep 11 18:02:52.217060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.217060    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0592ba-4253-4896-bf50-b427cbe39bc1" containerName="manager"
Sep 11 18:02:52.217282 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.217145    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b0592ba-4253-4896-bf50-b427cbe39bc1" containerName="manager"
Sep 11 18:02:52.220208 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.220186    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.222976 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.222953    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 18:02:52.223091 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.222988    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-622qz\""
Sep 11 18:02:52.228765 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.228741    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc54b69-92c8-4ed2-be6b-02651e084df7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.228878 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.228781    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.228878 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.228839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.228878 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.228864    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.229041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.228900    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5kst\" (UniqueName: \"kubernetes.io/projected/4dc54b69-92c8-4ed2-be6b-02651e084df7-kube-api-access-p5kst\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.229041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.228989    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.229136 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.229061    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.229174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.229128    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.229213 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.229187    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.231945 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.231924    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"]
Sep 11 18:02:52.330049 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330004    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330049 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330050    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330092    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330111    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc54b69-92c8-4ed2-be6b-02651e084df7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330134    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330155    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5kst\" (UniqueName: \"kubernetes.io/projected/4dc54b69-92c8-4ed2-be6b-02651e084df7-kube-api-access-p5kst\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330245    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330643 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330643 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330542    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330860 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330837    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.330860 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.331011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.330912    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc54b69-92c8-4ed2-be6b-02651e084df7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.332520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.332489    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.332726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.332708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.338569 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.338546    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc54b69-92c8-4ed2-be6b-02651e084df7-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.338700 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.338673    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5kst\" (UniqueName: \"kubernetes.io/projected/4dc54b69-92c8-4ed2-be6b-02651e084df7-kube-api-access-p5kst\") pod \"maas-default-gateway-openshift-default-5ccb968c66-v4qdh\" (UID: \"4dc54b69-92c8-4ed2-be6b-02651e084df7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.530600 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.530519    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:52.804501 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.804477    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"]
Sep 11 18:02:52.805525 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:52.805499    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc54b69_92c8_4ed2_be6b_02651e084df7.slice/crio-de54d2b14be2deefe0381e9ab3661973a280248d0d9dcd7ba4c53406044d4b41 WatchSource:0}: Error finding container de54d2b14be2deefe0381e9ab3661973a280248d0d9dcd7ba4c53406044d4b41: Status 404 returned error can't find the container with id de54d2b14be2deefe0381e9ab3661973a280248d0d9dcd7ba4c53406044d4b41
Sep 11 18:02:52.956255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:52.956220    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" event={"ID":"4dc54b69-92c8-4ed2-be6b-02651e084df7","Type":"ContainerStarted","Data":"de54d2b14be2deefe0381e9ab3661973a280248d0d9dcd7ba4c53406044d4b41"}
Sep 11 18:02:55.774076 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:55.774043    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:02:55.774341 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:55.774112    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:02:55.774341 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:55.774145    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:02:55.979170 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:55.979129    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" event={"ID":"4dc54b69-92c8-4ed2-be6b-02651e084df7","Type":"ContainerStarted","Data":"031a7065ab34539cb9df06659235ea8eceed0451343b7262b656886e1a89abbb"}
Sep 11 18:02:56.002177 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:56.002108    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" podStartSLOduration=1.035871087 podStartE2EDuration="4.002087931s" totalImagesPullingTime="2.966216844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:56.001635651 +0000 UTC m=+539.771702000" watchObservedRunningTime="2026-09-11 18:02:56.002087931 +0000 UTC m=+539.772154282"
Sep 11 18:02:56.530824 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:56.530793    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:56.535501 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:56.535476    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:56.985563 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:56.985529    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:56.986433 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:56.986413    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:02:57.033822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.033790    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xmwfj"]
Sep 11 18:02:57.038668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.038649    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:02:57.043518 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.043492    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 18:02:57.043675 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.043523    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dwv54\""
Sep 11 18:02:57.047113 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.047087    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xmwfj"]
Sep 11 18:02:57.069510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.069484    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfggc\" (UniqueName: \"kubernetes.io/projected/040a6c21-23a5-4a3f-b062-dddaffbf4a95-kube-api-access-lfggc\") pod \"limitador-limitador-8588749977-xmwfj\" (UID: \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\") " pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:02:57.069740 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.069720    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/040a6c21-23a5-4a3f-b062-dddaffbf4a95-config-file\") pod \"limitador-limitador-8588749977-xmwfj\" (UID: \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\") " pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:02:57.130338 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.130291    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xmwfj"]
Sep 11 18:02:57.170584 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.170557    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfggc\" (UniqueName: \"kubernetes.io/projected/040a6c21-23a5-4a3f-b062-dddaffbf4a95-kube-api-access-lfggc\") pod \"limitador-limitador-8588749977-xmwfj\" (UID: \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\") " pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:02:57.170753 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.170611    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/040a6c21-23a5-4a3f-b062-dddaffbf4a95-config-file\") pod \"limitador-limitador-8588749977-xmwfj\" (UID: \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\") " pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:02:57.171191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.171172    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/040a6c21-23a5-4a3f-b062-dddaffbf4a95-config-file\") pod \"limitador-limitador-8588749977-xmwfj\" (UID: \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\") " pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:02:57.179810 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.179788    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfggc\" (UniqueName: \"kubernetes.io/projected/040a6c21-23a5-4a3f-b062-dddaffbf4a95-kube-api-access-lfggc\") pod \"limitador-limitador-8588749977-xmwfj\" (UID: \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\") " pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:02:57.350732 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.350653    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:02:57.624729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.624704    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xmwfj"]
Sep 11 18:02:57.626467 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:57.626440    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040a6c21_23a5_4a3f_b062_dddaffbf4a95.slice/crio-04bf31c4fab199ad3ab3d1f8abe155dbe7732a8dfe2eb846d49d819feb294855 WatchSource:0}: Error finding container 04bf31c4fab199ad3ab3d1f8abe155dbe7732a8dfe2eb846d49d819feb294855: Status 404 returned error can't find the container with id 04bf31c4fab199ad3ab3d1f8abe155dbe7732a8dfe2eb846d49d819feb294855
Sep 11 18:02:57.845625 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.845594    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkg8v"]
Sep 11 18:02:57.850092 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.850072    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v"
Sep 11 18:02:57.853358 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.853340    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lk655\""
Sep 11 18:02:57.857033 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.857011    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkg8v"]
Sep 11 18:02:57.875141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.875076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndng7\" (UniqueName: \"kubernetes.io/projected/81f632fb-0027-49df-b0c2-fbfa1de630b7-kube-api-access-ndng7\") pod \"authorino-7bcfd8f9cd-hkg8v\" (UID: \"81f632fb-0027-49df-b0c2-fbfa1de630b7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v"
Sep 11 18:02:57.976160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.976120    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndng7\" (UniqueName: \"kubernetes.io/projected/81f632fb-0027-49df-b0c2-fbfa1de630b7-kube-api-access-ndng7\") pod \"authorino-7bcfd8f9cd-hkg8v\" (UID: \"81f632fb-0027-49df-b0c2-fbfa1de630b7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v"
Sep 11 18:02:57.987398 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.987374    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndng7\" (UniqueName: \"kubernetes.io/projected/81f632fb-0027-49df-b0c2-fbfa1de630b7-kube-api-access-ndng7\") pod \"authorino-7bcfd8f9cd-hkg8v\" (UID: \"81f632fb-0027-49df-b0c2-fbfa1de630b7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v"
Sep 11 18:02:57.991378 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.991326    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj" event={"ID":"040a6c21-23a5-4a3f-b062-dddaffbf4a95","Type":"ContainerStarted","Data":"04bf31c4fab199ad3ab3d1f8abe155dbe7732a8dfe2eb846d49d819feb294855"}
Sep 11 18:02:57.993281 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.993257    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-dmtqz"]
Sep 11 18:02:57.997050 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:57.997032    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dmtqz"
Sep 11 18:02:58.002039 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.002017    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dmtqz"]
Sep 11 18:02:58.077379 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.077344    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxlhc\" (UniqueName: \"kubernetes.io/projected/79c82003-0b3a-437e-8859-bd22ac689bbe-kube-api-access-fxlhc\") pod \"authorino-59c974bd48-dmtqz\" (UID: \"79c82003-0b3a-437e-8859-bd22ac689bbe\") " pod="kuadrant-system/authorino-59c974bd48-dmtqz"
Sep 11 18:02:58.161837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.161806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v"
Sep 11 18:02:58.178062 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.178031    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxlhc\" (UniqueName: \"kubernetes.io/projected/79c82003-0b3a-437e-8859-bd22ac689bbe-kube-api-access-fxlhc\") pod \"authorino-59c974bd48-dmtqz\" (UID: \"79c82003-0b3a-437e-8859-bd22ac689bbe\") " pod="kuadrant-system/authorino-59c974bd48-dmtqz"
Sep 11 18:02:58.187180 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.187155    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxlhc\" (UniqueName: \"kubernetes.io/projected/79c82003-0b3a-437e-8859-bd22ac689bbe-kube-api-access-fxlhc\") pod \"authorino-59c974bd48-dmtqz\" (UID: \"79c82003-0b3a-437e-8859-bd22ac689bbe\") " pod="kuadrant-system/authorino-59c974bd48-dmtqz"
Sep 11 18:02:58.306611 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.306576    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dmtqz"
Sep 11 18:02:58.430166 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.430144    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkg8v"]
Sep 11 18:02:58.431652 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:58.431626    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81f632fb_0027_49df_b0c2_fbfa1de630b7.slice/crio-dcc01fc34115b28a19c6e240d4451408e9f5fc2670177eaea52476f21bb27692 WatchSource:0}: Error finding container dcc01fc34115b28a19c6e240d4451408e9f5fc2670177eaea52476f21bb27692: Status 404 returned error can't find the container with id dcc01fc34115b28a19c6e240d4451408e9f5fc2670177eaea52476f21bb27692
Sep 11 18:02:58.575860 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.575830    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dmtqz"]
Sep 11 18:02:58.577235 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:02:58.577204    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79c82003_0b3a_437e_8859_bd22ac689bbe.slice/crio-059616ee8d137f70ed766c09ad9be6bb5418df2592160be36e07fc4f97cc1d1f WatchSource:0}: Error finding container 059616ee8d137f70ed766c09ad9be6bb5418df2592160be36e07fc4f97cc1d1f: Status 404 returned error can't find the container with id 059616ee8d137f70ed766c09ad9be6bb5418df2592160be36e07fc4f97cc1d1f
Sep 11 18:02:58.997919 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.997883    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v" event={"ID":"81f632fb-0027-49df-b0c2-fbfa1de630b7","Type":"ContainerStarted","Data":"dcc01fc34115b28a19c6e240d4451408e9f5fc2670177eaea52476f21bb27692"}
Sep 11 18:02:58.999174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:02:58.999149    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dmtqz" event={"ID":"79c82003-0b3a-437e-8859-bd22ac689bbe","Type":"ContainerStarted","Data":"059616ee8d137f70ed766c09ad9be6bb5418df2592160be36e07fc4f97cc1d1f"}
Sep 11 18:03:00.006820 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:00.006784    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj" event={"ID":"040a6c21-23a5-4a3f-b062-dddaffbf4a95","Type":"ContainerStarted","Data":"34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa"}
Sep 11 18:03:00.007325 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:00.006932    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:03:00.024889 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:00.024833    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj" podStartSLOduration=1.449244872 podStartE2EDuration="3.024818265s" totalImagesPullingTime="1.575573393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:00.022736269 +0000 UTC m=+543.792802617" watchObservedRunningTime="2026-09-11 18:03:00.024818265 +0000 UTC m=+543.794884613"
Sep 11 18:03:01.013503 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:01.013470    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v" event={"ID":"81f632fb-0027-49df-b0c2-fbfa1de630b7","Type":"ContainerStarted","Data":"9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b"}
Sep 11 18:03:01.015007 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:01.014981    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dmtqz" event={"ID":"79c82003-0b3a-437e-8859-bd22ac689bbe","Type":"ContainerStarted","Data":"8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af"}
Sep 11 18:03:01.031365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:01.031322    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v" podStartSLOduration=2.143847354 podStartE2EDuration="4.031308488s" totalImagesPullingTime="1.887461134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:01.029703206 +0000 UTC m=+544.799769553" watchObservedRunningTime="2026-09-11 18:03:01.031308488 +0000 UTC m=+544.801374833"
Sep 11 18:03:01.045690 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:01.045649    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-dmtqz" podStartSLOduration=2.30782069 podStartE2EDuration="4.045633884s" totalImagesPullingTime="1.737813194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:01.045222227 +0000 UTC m=+544.815288583" watchObservedRunningTime="2026-09-11 18:03:01.045633884 +0000 UTC m=+544.815700233"
Sep 11 18:03:01.075715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:01.075686    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkg8v"]
Sep 11 18:03:03.031999 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:03.031952    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v" podUID="81f632fb-0027-49df-b0c2-fbfa1de630b7" containerName="authorino" containerID="cri-o://9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b" gracePeriod=30
Sep 11 18:03:03.270463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:03.270440    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v"
Sep 11 18:03:03.322711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:03.322641    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndng7\" (UniqueName: \"kubernetes.io/projected/81f632fb-0027-49df-b0c2-fbfa1de630b7-kube-api-access-ndng7\") pod \"81f632fb-0027-49df-b0c2-fbfa1de630b7\" (UID: \"81f632fb-0027-49df-b0c2-fbfa1de630b7\") "
Sep 11 18:03:03.324569 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:03.324545    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f632fb-0027-49df-b0c2-fbfa1de630b7-kube-api-access-ndng7" (OuterVolumeSpecName: "kube-api-access-ndng7") pod "81f632fb-0027-49df-b0c2-fbfa1de630b7" (UID: "81f632fb-0027-49df-b0c2-fbfa1de630b7"). InnerVolumeSpecName "kube-api-access-ndng7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:03.424042 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:03.424011    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndng7\" (UniqueName: \"kubernetes.io/projected/81f632fb-0027-49df-b0c2-fbfa1de630b7-kube-api-access-ndng7\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:03:04.038050 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.038016    2571 generic.go:356] "Generic (PLEG): container finished" podID="81f632fb-0027-49df-b0c2-fbfa1de630b7" containerID="9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b" exitCode=0
Sep 11 18:03:04.038494 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.038078    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v"
Sep 11 18:03:04.038494 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.038113    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v" event={"ID":"81f632fb-0027-49df-b0c2-fbfa1de630b7","Type":"ContainerDied","Data":"9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b"}
Sep 11 18:03:04.038494 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.038161    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-hkg8v" event={"ID":"81f632fb-0027-49df-b0c2-fbfa1de630b7","Type":"ContainerDied","Data":"dcc01fc34115b28a19c6e240d4451408e9f5fc2670177eaea52476f21bb27692"}
Sep 11 18:03:04.038494 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.038180    2571 scope.go:117] "RemoveContainer" containerID="9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b"
Sep 11 18:03:04.047476 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.047457    2571 scope.go:117] "RemoveContainer" containerID="9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b"
Sep 11 18:03:04.047699 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:03:04.047682    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b\": container with ID starting with 9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b not found: ID does not exist" containerID="9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b"
Sep 11 18:03:04.047749 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.047709    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b"} err="failed to get container status \"9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b\": rpc error: code = NotFound desc = could not find container \"9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b\": container with ID starting with 9bc1f4a25de655ed811f0c4828c255bf9d136a49eb75b4267f60bce6f42a1f8b not found: ID does not exist"
Sep 11 18:03:04.059825 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.059806    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkg8v"]
Sep 11 18:03:04.063235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.063215    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-hkg8v"]
Sep 11 18:03:04.730249 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:04.730206    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f632fb-0027-49df-b0c2-fbfa1de630b7" path="/var/lib/kubelet/pods/81f632fb-0027-49df-b0c2-fbfa1de630b7/volumes"
Sep 11 18:03:11.015855 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:11.015824    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:03:12.334755 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:12.334716    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xmwfj"]
Sep 11 18:03:12.335203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:12.334957    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj" podUID="040a6c21-23a5-4a3f-b062-dddaffbf4a95" containerName="limitador" containerID="cri-o://34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa" gracePeriod=30
Sep 11 18:03:12.883043 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:12.883021    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:03:12.995162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:12.995125    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfggc\" (UniqueName: \"kubernetes.io/projected/040a6c21-23a5-4a3f-b062-dddaffbf4a95-kube-api-access-lfggc\") pod \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\" (UID: \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\") "
Sep 11 18:03:12.995383 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:12.995202    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/040a6c21-23a5-4a3f-b062-dddaffbf4a95-config-file\") pod \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\" (UID: \"040a6c21-23a5-4a3f-b062-dddaffbf4a95\") "
Sep 11 18:03:12.995609 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:12.995572    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040a6c21-23a5-4a3f-b062-dddaffbf4a95-config-file" (OuterVolumeSpecName: "config-file") pod "040a6c21-23a5-4a3f-b062-dddaffbf4a95" (UID: "040a6c21-23a5-4a3f-b062-dddaffbf4a95"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:03:12.997151 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:12.997130    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040a6c21-23a5-4a3f-b062-dddaffbf4a95-kube-api-access-lfggc" (OuterVolumeSpecName: "kube-api-access-lfggc") pod "040a6c21-23a5-4a3f-b062-dddaffbf4a95" (UID: "040a6c21-23a5-4a3f-b062-dddaffbf4a95"). InnerVolumeSpecName "kube-api-access-lfggc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:13.088339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.088279    2571 generic.go:356] "Generic (PLEG): container finished" podID="040a6c21-23a5-4a3f-b062-dddaffbf4a95" containerID="34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa" exitCode=0
Sep 11 18:03:13.088539 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.088336    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj" event={"ID":"040a6c21-23a5-4a3f-b062-dddaffbf4a95","Type":"ContainerDied","Data":"34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa"}
Sep 11 18:03:13.088539 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.088358    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj"
Sep 11 18:03:13.088539 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.088437    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xmwfj" event={"ID":"040a6c21-23a5-4a3f-b062-dddaffbf4a95","Type":"ContainerDied","Data":"04bf31c4fab199ad3ab3d1f8abe155dbe7732a8dfe2eb846d49d819feb294855"}
Sep 11 18:03:13.088539 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.088469    2571 scope.go:117] "RemoveContainer" containerID="34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa"
Sep 11 18:03:13.096443 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.096219    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfggc\" (UniqueName: \"kubernetes.io/projected/040a6c21-23a5-4a3f-b062-dddaffbf4a95-kube-api-access-lfggc\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:03:13.096443 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.096246    2571 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/040a6c21-23a5-4a3f-b062-dddaffbf4a95-config-file\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:03:13.098093 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.098076    2571 scope.go:117] "RemoveContainer" containerID="34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa"
Sep 11 18:03:13.098404 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:03:13.098381    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa\": container with ID starting with 34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa not found: ID does not exist" containerID="34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa"
Sep 11 18:03:13.098460 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.098417    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa"} err="failed to get container status \"34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa\": rpc error: code = NotFound desc = could not find container \"34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa\": container with ID starting with 34c81936303353193a5fd1088ecb776c41b58dcf0c98c06bbbb5adfd0b63e7fa not found: ID does not exist"
Sep 11 18:03:13.113056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.113032    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xmwfj"]
Sep 11 18:03:13.118158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:13.118137    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xmwfj"]
Sep 11 18:03:14.730149 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:14.730118    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="040a6c21-23a5-4a3f-b062-dddaffbf4a95" path="/var/lib/kubelet/pods/040a6c21-23a5-4a3f-b062-dddaffbf4a95/volumes"
Sep 11 18:03:18.704768 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.704733    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"]
Sep 11 18:03:18.705229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.705087    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="040a6c21-23a5-4a3f-b062-dddaffbf4a95" containerName="limitador"
Sep 11 18:03:18.705229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.705099    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="040a6c21-23a5-4a3f-b062-dddaffbf4a95" containerName="limitador"
Sep 11 18:03:18.705229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.705121    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f632fb-0027-49df-b0c2-fbfa1de630b7" containerName="authorino"
Sep 11 18:03:18.705229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.705127    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f632fb-0027-49df-b0c2-fbfa1de630b7" containerName="authorino"
Sep 11 18:03:18.705229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.705179    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f632fb-0027-49df-b0c2-fbfa1de630b7" containerName="authorino"
Sep 11 18:03:18.705229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.705190    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="040a6c21-23a5-4a3f-b062-dddaffbf4a95" containerName="limitador"
Sep 11 18:03:18.709344 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.709323    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:18.712330 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.712283    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:03:18.712448 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.712353    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 18:03:18.712596 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.712579    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-lb8bm\""
Sep 11 18:03:18.713541 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.713525    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:03:18.717845 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.717800    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"]
Sep 11 18:03:18.842054 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.842020    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4b7a9066-b3da-4af9-8956-9c3d78c30f4d-data\") pod \"postgres-69fb4594cf-ktwm6\" (UID: \"4b7a9066-b3da-4af9-8956-9c3d78c30f4d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:18.842345 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.842324    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwhf\" (UniqueName: \"kubernetes.io/projected/4b7a9066-b3da-4af9-8956-9c3d78c30f4d-kube-api-access-7jwhf\") pod \"postgres-69fb4594cf-ktwm6\" (UID: \"4b7a9066-b3da-4af9-8956-9c3d78c30f4d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:18.943517 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.943486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4b7a9066-b3da-4af9-8956-9c3d78c30f4d-data\") pod \"postgres-69fb4594cf-ktwm6\" (UID: \"4b7a9066-b3da-4af9-8956-9c3d78c30f4d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:18.943673 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.943549    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwhf\" (UniqueName: \"kubernetes.io/projected/4b7a9066-b3da-4af9-8956-9c3d78c30f4d-kube-api-access-7jwhf\") pod \"postgres-69fb4594cf-ktwm6\" (UID: \"4b7a9066-b3da-4af9-8956-9c3d78c30f4d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:18.943875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.943853    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4b7a9066-b3da-4af9-8956-9c3d78c30f4d-data\") pod \"postgres-69fb4594cf-ktwm6\" (UID: \"4b7a9066-b3da-4af9-8956-9c3d78c30f4d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:18.952354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:18.952332    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwhf\" (UniqueName: \"kubernetes.io/projected/4b7a9066-b3da-4af9-8956-9c3d78c30f4d-kube-api-access-7jwhf\") pod \"postgres-69fb4594cf-ktwm6\" (UID: \"4b7a9066-b3da-4af9-8956-9c3d78c30f4d\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:19.023169 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:19.023104    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:19.293067 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:19.293040    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"]
Sep 11 18:03:19.294213 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:03:19.294188    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7a9066_b3da_4af9_8956_9c3d78c30f4d.slice/crio-a43c1fdfb864e289a4a6d2bed609c6a039d0c94d5b38294a28205d1924dcfb6f WatchSource:0}: Error finding container a43c1fdfb864e289a4a6d2bed609c6a039d0c94d5b38294a28205d1924dcfb6f: Status 404 returned error can't find the container with id a43c1fdfb864e289a4a6d2bed609c6a039d0c94d5b38294a28205d1924dcfb6f
Sep 11 18:03:20.126524 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:20.126486    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6" event={"ID":"4b7a9066-b3da-4af9-8956-9c3d78c30f4d","Type":"ContainerStarted","Data":"a43c1fdfb864e289a4a6d2bed609c6a039d0c94d5b38294a28205d1924dcfb6f"}
Sep 11 18:03:25.158612 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:25.158578    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6" event={"ID":"4b7a9066-b3da-4af9-8956-9c3d78c30f4d","Type":"ContainerStarted","Data":"3ce7fadd43a4fc2c4f2ea705fd10b4fc2ea1d8b6322ea87d7c732762b3794dd6"}
Sep 11 18:03:25.158993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:25.158678    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:25.176838 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:25.176786    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6" podStartSLOduration=2.275396806 podStartE2EDuration="7.176769997s" totalImagesPullingTime="4.901373191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:25.175484605 +0000 UTC m=+568.945550952" watchObservedRunningTime="2026-09-11 18:03:25.176769997 +0000 UTC m=+568.946836345"
Sep 11 18:03:31.192346 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:31.192291    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-ktwm6"
Sep 11 18:03:31.816183 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:31.816147    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-24d6c"]
Sep 11 18:03:31.820404 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:31.820388    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-24d6c"
Sep 11 18:03:31.837740 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:31.837716    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-24d6c"]
Sep 11 18:03:31.960717 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:31.960681    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-24d6c"]
Sep 11 18:03:31.960717 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:31.960698    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7cf\" (UniqueName: \"kubernetes.io/projected/a9e0223c-1a83-4045-86b1-44e9a29eb3c7-kube-api-access-rp7cf\") pod \"authorino-557f876f69-24d6c\" (UID: \"a9e0223c-1a83-4045-86b1-44e9a29eb3c7\") " pod="kuadrant-system/authorino-557f876f69-24d6c"
Sep 11 18:03:31.960903 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:03:31.960885    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rp7cf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-24d6c" podUID="a9e0223c-1a83-4045-86b1-44e9a29eb3c7"
Sep 11 18:03:31.996467 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:31.996442    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ltdpk"]
Sep 11 18:03:32.000482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.000460    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.008398 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.008376    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 18:03:32.010777 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.010759    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ltdpk"]
Sep 11 18:03:32.061468 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.061440    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7cf\" (UniqueName: \"kubernetes.io/projected/a9e0223c-1a83-4045-86b1-44e9a29eb3c7-kube-api-access-rp7cf\") pod \"authorino-557f876f69-24d6c\" (UID: \"a9e0223c-1a83-4045-86b1-44e9a29eb3c7\") " pod="kuadrant-system/authorino-557f876f69-24d6c"
Sep 11 18:03:32.071824 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.071762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7cf\" (UniqueName: \"kubernetes.io/projected/a9e0223c-1a83-4045-86b1-44e9a29eb3c7-kube-api-access-rp7cf\") pod \"authorino-557f876f69-24d6c\" (UID: \"a9e0223c-1a83-4045-86b1-44e9a29eb3c7\") " pod="kuadrant-system/authorino-557f876f69-24d6c"
Sep 11 18:03:32.078205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.078176    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ltdpk"]
Sep 11 18:03:32.078517 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:03:32.078493    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nsvzn tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-nsvzn tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-ltdpk" podUID="ebabf491-fe71-4a30-99e0-c93df9df8b8d"
Sep 11 18:03:32.111496 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.111467    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-767c56875-kd8wx"]
Sep 11 18:03:32.115858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.115842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:03:32.125428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.125406    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-767c56875-kd8wx"]
Sep 11 18:03:32.162655 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.162626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsvzn\" (UniqueName: \"kubernetes.io/projected/ebabf491-fe71-4a30-99e0-c93df9df8b8d-kube-api-access-nsvzn\") pod \"authorino-6d4dfc649f-ltdpk\" (UID: \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\") " pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.162807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.162676    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ebabf491-fe71-4a30-99e0-c93df9df8b8d-tls-cert\") pod \"authorino-6d4dfc649f-ltdpk\" (UID: \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\") " pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.193415 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.193386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.193807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.193386    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-24d6c"
Sep 11 18:03:32.201744 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.201726    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.205012 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.204997    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-24d6c"
Sep 11 18:03:32.263682 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.263656    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-tls-cert\") pod \"authorino-767c56875-kd8wx\" (UID: \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\") " pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:03:32.263835 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.263688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgn4r\" (UniqueName: \"kubernetes.io/projected/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-kube-api-access-qgn4r\") pod \"authorino-767c56875-kd8wx\" (UID: \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\") " pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:03:32.263835 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.263712    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ebabf491-fe71-4a30-99e0-c93df9df8b8d-tls-cert\") pod \"authorino-6d4dfc649f-ltdpk\" (UID: \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\") " pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.263835 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.263811    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsvzn\" (UniqueName: \"kubernetes.io/projected/ebabf491-fe71-4a30-99e0-c93df9df8b8d-kube-api-access-nsvzn\") pod \"authorino-6d4dfc649f-ltdpk\" (UID: \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\") " pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.266070 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.266049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ebabf491-fe71-4a30-99e0-c93df9df8b8d-tls-cert\") pod \"authorino-6d4dfc649f-ltdpk\" (UID: \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\") " pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.273086 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.273065    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsvzn\" (UniqueName: \"kubernetes.io/projected/ebabf491-fe71-4a30-99e0-c93df9df8b8d-kube-api-access-nsvzn\") pod \"authorino-6d4dfc649f-ltdpk\" (UID: \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\") " pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:32.364287 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.364208    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsvzn\" (UniqueName: \"kubernetes.io/projected/ebabf491-fe71-4a30-99e0-c93df9df8b8d-kube-api-access-nsvzn\") pod \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\" (UID: \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\") "
Sep 11 18:03:32.364287 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.364243    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ebabf491-fe71-4a30-99e0-c93df9df8b8d-tls-cert\") pod \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\" (UID: \"ebabf491-fe71-4a30-99e0-c93df9df8b8d\") "
Sep 11 18:03:32.364489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.364312    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp7cf\" (UniqueName: \"kubernetes.io/projected/a9e0223c-1a83-4045-86b1-44e9a29eb3c7-kube-api-access-rp7cf\") pod \"a9e0223c-1a83-4045-86b1-44e9a29eb3c7\" (UID: \"a9e0223c-1a83-4045-86b1-44e9a29eb3c7\") "
Sep 11 18:03:32.364542 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.364524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-tls-cert\") pod \"authorino-767c56875-kd8wx\" (UID: \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\") " pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:03:32.364578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.364564    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgn4r\" (UniqueName: \"kubernetes.io/projected/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-kube-api-access-qgn4r\") pod \"authorino-767c56875-kd8wx\" (UID: \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\") " pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:03:32.366486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.366461    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebabf491-fe71-4a30-99e0-c93df9df8b8d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ebabf491-fe71-4a30-99e0-c93df9df8b8d" (UID: "ebabf491-fe71-4a30-99e0-c93df9df8b8d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:03:32.366486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.366477    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e0223c-1a83-4045-86b1-44e9a29eb3c7-kube-api-access-rp7cf" (OuterVolumeSpecName: "kube-api-access-rp7cf") pod "a9e0223c-1a83-4045-86b1-44e9a29eb3c7" (UID: "a9e0223c-1a83-4045-86b1-44e9a29eb3c7"). InnerVolumeSpecName "kube-api-access-rp7cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:32.366622 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.366492    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebabf491-fe71-4a30-99e0-c93df9df8b8d-kube-api-access-nsvzn" (OuterVolumeSpecName: "kube-api-access-nsvzn") pod "ebabf491-fe71-4a30-99e0-c93df9df8b8d" (UID: "ebabf491-fe71-4a30-99e0-c93df9df8b8d"). InnerVolumeSpecName "kube-api-access-nsvzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:32.366819 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.366804    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-tls-cert\") pod \"authorino-767c56875-kd8wx\" (UID: \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\") " pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:03:32.373288 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.373258    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgn4r\" (UniqueName: \"kubernetes.io/projected/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-kube-api-access-qgn4r\") pod \"authorino-767c56875-kd8wx\" (UID: \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\") " pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:03:32.425724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.425700    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:03:32.465120 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.465091    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsvzn\" (UniqueName: \"kubernetes.io/projected/ebabf491-fe71-4a30-99e0-c93df9df8b8d-kube-api-access-nsvzn\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:03:32.465120 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.465118    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ebabf491-fe71-4a30-99e0-c93df9df8b8d-tls-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:03:32.465346 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.465131    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp7cf\" (UniqueName: \"kubernetes.io/projected/a9e0223c-1a83-4045-86b1-44e9a29eb3c7-kube-api-access-rp7cf\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:03:32.695107 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:32.695079    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-767c56875-kd8wx"]
Sep 11 18:03:32.696764 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:03:32.696740    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46a7d14b_6c29_4cfd_88b2_0f9ac930906f.slice/crio-2f98ec687ffcfbe59670dc91cc8d55478a60f9b15307acc3de4536b3a0470773 WatchSource:0}: Error finding container 2f98ec687ffcfbe59670dc91cc8d55478a60f9b15307acc3de4536b3a0470773: Status 404 returned error can't find the container with id 2f98ec687ffcfbe59670dc91cc8d55478a60f9b15307acc3de4536b3a0470773
Sep 11 18:03:33.199485 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:33.199455    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-24d6c"
Sep 11 18:03:33.199846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:33.199458    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-767c56875-kd8wx" event={"ID":"46a7d14b-6c29-4cfd-88b2-0f9ac930906f","Type":"ContainerStarted","Data":"2f98ec687ffcfbe59670dc91cc8d55478a60f9b15307acc3de4536b3a0470773"}
Sep 11 18:03:33.199846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:33.199634    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-ltdpk"
Sep 11 18:03:33.228419 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:33.228390    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ltdpk"]
Sep 11 18:03:33.232238 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:33.232213    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-ltdpk"]
Sep 11 18:03:33.254640 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:33.254615    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-24d6c"]
Sep 11 18:03:33.259739 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:33.259714    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-24d6c"]
Sep 11 18:03:34.205916 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.205879    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-767c56875-kd8wx" event={"ID":"46a7d14b-6c29-4cfd-88b2-0f9ac930906f","Type":"ContainerStarted","Data":"d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7"}
Sep 11 18:03:34.223794 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.223751    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-767c56875-kd8wx" podStartSLOduration=1.04566572 podStartE2EDuration="2.223738301s" totalImagesPullingTime="1.178072581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:34.221121796 +0000 UTC m=+577.991188144" watchObservedRunningTime="2026-09-11 18:03:34.223738301 +0000 UTC m=+577.993804648"
Sep 11 18:03:34.247848 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.247822    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dmtqz"]
Sep 11 18:03:34.248006 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.247987    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-dmtqz" podUID="79c82003-0b3a-437e-8859-bd22ac689bbe" containerName="authorino" containerID="cri-o://8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af" gracePeriod=30
Sep 11 18:03:34.504833 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.504811    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dmtqz"
Sep 11 18:03:34.580210 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.580171    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxlhc\" (UniqueName: \"kubernetes.io/projected/79c82003-0b3a-437e-8859-bd22ac689bbe-kube-api-access-fxlhc\") pod \"79c82003-0b3a-437e-8859-bd22ac689bbe\" (UID: \"79c82003-0b3a-437e-8859-bd22ac689bbe\") "
Sep 11 18:03:34.582234 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.582209    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c82003-0b3a-437e-8859-bd22ac689bbe-kube-api-access-fxlhc" (OuterVolumeSpecName: "kube-api-access-fxlhc") pod "79c82003-0b3a-437e-8859-bd22ac689bbe" (UID: "79c82003-0b3a-437e-8859-bd22ac689bbe"). InnerVolumeSpecName "kube-api-access-fxlhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:03:34.681616 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.681581    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxlhc\" (UniqueName: \"kubernetes.io/projected/79c82003-0b3a-437e-8859-bd22ac689bbe-kube-api-access-fxlhc\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:03:34.730357 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.730329    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e0223c-1a83-4045-86b1-44e9a29eb3c7" path="/var/lib/kubelet/pods/a9e0223c-1a83-4045-86b1-44e9a29eb3c7/volumes"
Sep 11 18:03:34.730626 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:34.730609    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebabf491-fe71-4a30-99e0-c93df9df8b8d" path="/var/lib/kubelet/pods/ebabf491-fe71-4a30-99e0-c93df9df8b8d/volumes"
Sep 11 18:03:35.212835 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.212801    2571 generic.go:356] "Generic (PLEG): container finished" podID="79c82003-0b3a-437e-8859-bd22ac689bbe" containerID="8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af" exitCode=0
Sep 11 18:03:35.213340 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.212854    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-dmtqz"
Sep 11 18:03:35.213340 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.212886    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dmtqz" event={"ID":"79c82003-0b3a-437e-8859-bd22ac689bbe","Type":"ContainerDied","Data":"8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af"}
Sep 11 18:03:35.213340 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.212919    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-dmtqz" event={"ID":"79c82003-0b3a-437e-8859-bd22ac689bbe","Type":"ContainerDied","Data":"059616ee8d137f70ed766c09ad9be6bb5418df2592160be36e07fc4f97cc1d1f"}
Sep 11 18:03:35.213340 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.212931    2571 scope.go:117] "RemoveContainer" containerID="8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af"
Sep 11 18:03:35.221565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.221547    2571 scope.go:117] "RemoveContainer" containerID="8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af"
Sep 11 18:03:35.221810 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:03:35.221792    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af\": container with ID starting with 8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af not found: ID does not exist" containerID="8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af"
Sep 11 18:03:35.221864 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.221819    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af"} err="failed to get container status \"8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af\": rpc error: code = NotFound desc = could not find container \"8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af\": container with ID starting with 8175f14a05a10671025f5efc04ce3ebf90a818d0bba3ee352357c67550ff32af not found: ID does not exist"
Sep 11 18:03:35.232517 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.232496    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dmtqz"]
Sep 11 18:03:35.236489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:35.236470    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-dmtqz"]
Sep 11 18:03:36.735884 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:03:36.735848    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c82003-0b3a-437e-8859-bd22ac689bbe" path="/var/lib/kubelet/pods/79c82003-0b3a-437e-8859-bd22ac689bbe/volumes"
Sep 11 18:04:15.152950 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.152918    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"]
Sep 11 18:04:15.155473 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.153249    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c82003-0b3a-437e-8859-bd22ac689bbe" containerName="authorino"
Sep 11 18:04:15.155473 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.153260    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c82003-0b3a-437e-8859-bd22ac689bbe" containerName="authorino"
Sep 11 18:04:15.155473 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.153339    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c82003-0b3a-437e-8859-bd22ac689bbe" containerName="authorino"
Sep 11 18:04:15.156242 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.156227    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:15.159406 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.159387    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 18:04:15.159902 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.159887    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-4r4rh\""
Sep 11 18:04:15.167436 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.167412    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"]
Sep 11 18:04:15.178017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.177995    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7q2\" (UniqueName: \"kubernetes.io/projected/34cfa5cb-dbc3-4c99-903e-944f864cef32-kube-api-access-rr7q2\") pod \"payload-processing-5b54b7fbb7-g2ncf\" (UID: \"34cfa5cb-dbc3-4c99-903e-944f864cef32\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:15.178135 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.178026    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34cfa5cb-dbc3-4c99-903e-944f864cef32-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-g2ncf\" (UID: \"34cfa5cb-dbc3-4c99-903e-944f864cef32\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:15.278745 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.278706    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7q2\" (UniqueName: \"kubernetes.io/projected/34cfa5cb-dbc3-4c99-903e-944f864cef32-kube-api-access-rr7q2\") pod \"payload-processing-5b54b7fbb7-g2ncf\" (UID: \"34cfa5cb-dbc3-4c99-903e-944f864cef32\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:15.278745 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.278747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34cfa5cb-dbc3-4c99-903e-944f864cef32-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-g2ncf\" (UID: \"34cfa5cb-dbc3-4c99-903e-944f864cef32\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:15.279424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.279404    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34cfa5cb-dbc3-4c99-903e-944f864cef32-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-g2ncf\" (UID: \"34cfa5cb-dbc3-4c99-903e-944f864cef32\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:15.297566 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.297531    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7q2\" (UniqueName: \"kubernetes.io/projected/34cfa5cb-dbc3-4c99-903e-944f864cef32-kube-api-access-rr7q2\") pod \"payload-processing-5b54b7fbb7-g2ncf\" (UID: \"34cfa5cb-dbc3-4c99-903e-944f864cef32\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:15.466478 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.466442    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:15.776857 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:04:15.776818    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cfa5cb_dbc3_4c99_903e_944f864cef32.slice/crio-06d7550efca3a949efc628e0312ac5c0e83f361bc2f714ff31159b36dd087ac9 WatchSource:0}: Error finding container 06d7550efca3a949efc628e0312ac5c0e83f361bc2f714ff31159b36dd087ac9: Status 404 returned error can't find the container with id 06d7550efca3a949efc628e0312ac5c0e83f361bc2f714ff31159b36dd087ac9
Sep 11 18:04:15.778225 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.778200    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"]
Sep 11 18:04:15.778877 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:15.778860    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:04:16.433005 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:16.432957    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf" event={"ID":"34cfa5cb-dbc3-4c99-903e-944f864cef32","Type":"ContainerStarted","Data":"06d7550efca3a949efc628e0312ac5c0e83f361bc2f714ff31159b36dd087ac9"}
Sep 11 18:04:18.446338 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:18.446283    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf" event={"ID":"34cfa5cb-dbc3-4c99-903e-944f864cef32","Type":"ContainerStarted","Data":"6ae8137c9d668032b1f90e367130f7a974c1b21401a36ac34a77d6cc4fc3b2f4"}
Sep 11 18:04:18.446736 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:18.446443    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:18.468782 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:18.468716    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf" podStartSLOduration=1.470232033 podStartE2EDuration="3.468696042s" totalImagesPullingTime="1.998464009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:18.467778371 +0000 UTC m=+622.237844719" watchObservedRunningTime="2026-09-11 18:04:18.468696042 +0000 UTC m=+622.238762393"
Sep 11 18:04:24.484597 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:24.484565    2571 generic.go:356] "Generic (PLEG): container finished" podID="4dc54b69-92c8-4ed2-be6b-02651e084df7" containerID="031a7065ab34539cb9df06659235ea8eceed0451343b7262b656886e1a89abbb" exitCode=0
Sep 11 18:04:24.484959 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:24.484633    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" event={"ID":"4dc54b69-92c8-4ed2-be6b-02651e084df7","Type":"ContainerDied","Data":"031a7065ab34539cb9df06659235ea8eceed0451343b7262b656886e1a89abbb"}
Sep 11 18:04:24.485047 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:24.485031    2571 scope.go:117] "RemoveContainer" containerID="031a7065ab34539cb9df06659235ea8eceed0451343b7262b656886e1a89abbb"
Sep 11 18:04:24.486018 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:24.485983    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:04:24.486082 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:24.486070    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:04:24.486121 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:24.486102    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:04:25.494551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:25.494519    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" event={"ID":"4dc54b69-92c8-4ed2-be6b-02651e084df7","Type":"ContainerStarted","Data":"c82e11570b1b958797b6789811448bbb54592afaad689643743b75e791c0442a"}
Sep 11 18:04:25.531282 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:25.531253    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:04:25.532045 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:25.532020    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v4qdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 18:04:25.532190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:25.532071    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" podUID="4dc54b69-92c8-4ed2-be6b-02651e084df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 18:04:26.531066 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:26.531029    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v4qdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 18:04:26.531549 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:26.531088    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" podUID="4dc54b69-92c8-4ed2-be6b-02651e084df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 18:04:26.985722 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:26.985681    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:04:27.531503 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:27.531472    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v4qdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 18:04:27.531883 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:27.531527    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" podUID="4dc54b69-92c8-4ed2-be6b-02651e084df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 18:04:28.531243 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:28.531203    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v4qdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 18:04:28.531518 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:28.531272    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" podUID="4dc54b69-92c8-4ed2-be6b-02651e084df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 18:04:29.456493 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:29.456459    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-g2ncf"
Sep 11 18:04:29.531565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:29.531532    2571 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-v4qdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 11 18:04:29.531931 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:29.531582    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh" podUID="4dc54b69-92c8-4ed2-be6b-02651e084df7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 11 18:04:30.536123 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:30.536094    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:04:30.537202 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:30.537181    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-v4qdh"
Sep 11 18:04:33.523511 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.523480    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"]
Sep 11 18:04:33.530101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.530076    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.532953 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.532933    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:04:33.534073 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.534058    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:04:33.534073 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.534064    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d5n2t\""
Sep 11 18:04:33.534244 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.534110    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 18:04:33.539056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.539031    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"]
Sep 11 18:04:33.636083 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.636045    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.636083 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.636088    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.636293 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.636121    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.636293 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.636192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0f6b98-85bf-4087-aae4-6455f5af7568-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.636293 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.636219    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.636293 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.636254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czldn\" (UniqueName: \"kubernetes.io/projected/dd0f6b98-85bf-4087-aae4-6455f5af7568-kube-api-access-czldn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737157 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737116    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0f6b98-85bf-4087-aae4-6455f5af7568-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737157 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737153    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737402 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737172    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czldn\" (UniqueName: \"kubernetes.io/projected/dd0f6b98-85bf-4087-aae4-6455f5af7568-kube-api-access-czldn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737402 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737210    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737402 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737235    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737402 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737271    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737644 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737705 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737651    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.737705 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.737696    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.739652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.739632    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd0f6b98-85bf-4087-aae4-6455f5af7568-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.739761 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.739732    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd0f6b98-85bf-4087-aae4-6455f5af7568-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.745783 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.745753    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czldn\" (UniqueName: \"kubernetes.io/projected/dd0f6b98-85bf-4087-aae4-6455f5af7568-kube-api-access-czldn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj\" (UID: \"dd0f6b98-85bf-4087-aae4-6455f5af7568\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:33.841391 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:33.841290    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:34.122410 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:34.122375    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"]
Sep 11 18:04:34.124048 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:04:34.124018    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0f6b98_85bf_4087_aae4_6455f5af7568.slice/crio-a3e9dc3349a0892035c03fa362353f071781cf1a6685274eeb319dc10b736ea3 WatchSource:0}: Error finding container a3e9dc3349a0892035c03fa362353f071781cf1a6685274eeb319dc10b736ea3: Status 404 returned error can't find the container with id a3e9dc3349a0892035c03fa362353f071781cf1a6685274eeb319dc10b736ea3
Sep 11 18:04:34.546034 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:34.545992    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj" event={"ID":"dd0f6b98-85bf-4087-aae4-6455f5af7568","Type":"ContainerStarted","Data":"a3e9dc3349a0892035c03fa362353f071781cf1a6685274eeb319dc10b736ea3"}
Sep 11 18:04:35.553181 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:35.553140    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj" event={"ID":"dd0f6b98-85bf-4087-aae4-6455f5af7568","Type":"ContainerStarted","Data":"8180c3052c83d5dad124da0647f7c7ce481be57cdd7fbba583ae1f5c461c755a"}
Sep 11 18:04:35.573016 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:35.572949    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj" podStartSLOduration=1.423569355 podStartE2EDuration="2.572927665s" totalImagesPullingTime="1.14935831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:04:35.572384986 +0000 UTC m=+639.342451335" watchObservedRunningTime="2026-09-11 18:04:35.572927665 +0000 UTC m=+639.342994016"
Sep 11 18:04:43.841875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:43.841837    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:43.842249 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:43.841919    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:43.854393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:43.854371    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:04:44.610688 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:04:44.610660    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj"
Sep 11 18:06:56.226177 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.226142    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b7dcf54b-kvs4q"]
Sep 11 18:06:56.229545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.229525    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7dcf54b-kvs4q"
Sep 11 18:06:56.248828 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.248801    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b7dcf54b-kvs4q"]
Sep 11 18:06:56.314375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.314338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqbq7\" (UniqueName: \"kubernetes.io/projected/fd1aa1c1-771d-4055-b34a-b501595e51d4-kube-api-access-fqbq7\") pod \"authorino-6b7dcf54b-kvs4q\" (UID: \"fd1aa1c1-771d-4055-b34a-b501595e51d4\") " pod="kuadrant-system/authorino-6b7dcf54b-kvs4q"
Sep 11 18:06:56.314520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.314391    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd1aa1c1-771d-4055-b34a-b501595e51d4-tls-cert\") pod \"authorino-6b7dcf54b-kvs4q\" (UID: \"fd1aa1c1-771d-4055-b34a-b501595e51d4\") " pod="kuadrant-system/authorino-6b7dcf54b-kvs4q"
Sep 11 18:06:56.415663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.415631    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbq7\" (UniqueName: \"kubernetes.io/projected/fd1aa1c1-771d-4055-b34a-b501595e51d4-kube-api-access-fqbq7\") pod \"authorino-6b7dcf54b-kvs4q\" (UID: \"fd1aa1c1-771d-4055-b34a-b501595e51d4\") " pod="kuadrant-system/authorino-6b7dcf54b-kvs4q"
Sep 11 18:06:56.415826 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.415674    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd1aa1c1-771d-4055-b34a-b501595e51d4-tls-cert\") pod \"authorino-6b7dcf54b-kvs4q\" (UID: \"fd1aa1c1-771d-4055-b34a-b501595e51d4\") " pod="kuadrant-system/authorino-6b7dcf54b-kvs4q"
Sep 11 18:06:56.418026 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.417999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fd1aa1c1-771d-4055-b34a-b501595e51d4-tls-cert\") pod \"authorino-6b7dcf54b-kvs4q\" (UID: \"fd1aa1c1-771d-4055-b34a-b501595e51d4\") " pod="kuadrant-system/authorino-6b7dcf54b-kvs4q"
Sep 11 18:06:56.427388 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.427366    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqbq7\" (UniqueName: \"kubernetes.io/projected/fd1aa1c1-771d-4055-b34a-b501595e51d4-kube-api-access-fqbq7\") pod \"authorino-6b7dcf54b-kvs4q\" (UID: \"fd1aa1c1-771d-4055-b34a-b501595e51d4\") " pod="kuadrant-system/authorino-6b7dcf54b-kvs4q"
Sep 11 18:06:56.538679 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.538596    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b7dcf54b-kvs4q"
Sep 11 18:06:56.812451 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:56.812430    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b7dcf54b-kvs4q"]
Sep 11 18:06:56.813972 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:06:56.813938    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd1aa1c1_771d_4055_b34a_b501595e51d4.slice/crio-9907ad614896abadeccd3226c570da0cb5420ed7d5f5e35387ce1295962449ca WatchSource:0}: Error finding container 9907ad614896abadeccd3226c570da0cb5420ed7d5f5e35387ce1295962449ca: Status 404 returned error can't find the container with id 9907ad614896abadeccd3226c570da0cb5420ed7d5f5e35387ce1295962449ca
Sep 11 18:06:57.311329 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:57.311279    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7dcf54b-kvs4q" event={"ID":"fd1aa1c1-771d-4055-b34a-b501595e51d4","Type":"ContainerStarted","Data":"9907ad614896abadeccd3226c570da0cb5420ed7d5f5e35387ce1295962449ca"}
Sep 11 18:06:58.317926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.317881    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b7dcf54b-kvs4q" event={"ID":"fd1aa1c1-771d-4055-b34a-b501595e51d4","Type":"ContainerStarted","Data":"f3ae96fff77abf0b11c89bbb2e3ef36fe929fb9dc0c171a529da84faf5181ff8"}
Sep 11 18:06:58.346869 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.346804    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b7dcf54b-kvs4q" podStartSLOduration=1.622894058 podStartE2EDuration="2.346781233s" totalImagesPullingTime="723.887175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:06:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:06:58.344350927 +0000 UTC m=+782.114417270" watchObservedRunningTime="2026-09-11 18:06:58.346781233 +0000 UTC m=+782.116847582"
Sep 11 18:06:58.427327 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.427281    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-767c56875-kd8wx"]
Sep 11 18:06:58.427500 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.427474    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-767c56875-kd8wx" podUID="46a7d14b-6c29-4cfd-88b2-0f9ac930906f" containerName="authorino" containerID="cri-o://d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7" gracePeriod=30
Sep 11 18:06:58.705227 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.705199    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:06:58.838080 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.838044    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-tls-cert\") pod \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\" (UID: \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\") "
Sep 11 18:06:58.838264 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.838113    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgn4r\" (UniqueName: \"kubernetes.io/projected/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-kube-api-access-qgn4r\") pod \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\" (UID: \"46a7d14b-6c29-4cfd-88b2-0f9ac930906f\") "
Sep 11 18:06:58.840373 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.840336    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-kube-api-access-qgn4r" (OuterVolumeSpecName: "kube-api-access-qgn4r") pod "46a7d14b-6c29-4cfd-88b2-0f9ac930906f" (UID: "46a7d14b-6c29-4cfd-88b2-0f9ac930906f"). InnerVolumeSpecName "kube-api-access-qgn4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:06:58.848579 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.848549    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "46a7d14b-6c29-4cfd-88b2-0f9ac930906f" (UID: "46a7d14b-6c29-4cfd-88b2-0f9ac930906f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:06:58.939344 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.939289    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-tls-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:06:58.939503 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:58.939349    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgn4r\" (UniqueName: \"kubernetes.io/projected/46a7d14b-6c29-4cfd-88b2-0f9ac930906f-kube-api-access-qgn4r\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:06:59.324908 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.324827    2571 generic.go:356] "Generic (PLEG): container finished" podID="46a7d14b-6c29-4cfd-88b2-0f9ac930906f" containerID="d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7" exitCode=0
Sep 11 18:06:59.325316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.324911    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-767c56875-kd8wx"
Sep 11 18:06:59.325316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.324915    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-767c56875-kd8wx" event={"ID":"46a7d14b-6c29-4cfd-88b2-0f9ac930906f","Type":"ContainerDied","Data":"d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7"}
Sep 11 18:06:59.325316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.325023    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-767c56875-kd8wx" event={"ID":"46a7d14b-6c29-4cfd-88b2-0f9ac930906f","Type":"ContainerDied","Data":"2f98ec687ffcfbe59670dc91cc8d55478a60f9b15307acc3de4536b3a0470773"}
Sep 11 18:06:59.325316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.325041    2571 scope.go:117] "RemoveContainer" containerID="d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7"
Sep 11 18:06:59.334244 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.334227    2571 scope.go:117] "RemoveContainer" containerID="d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7"
Sep 11 18:06:59.334541 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:06:59.334509    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7\": container with ID starting with d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7 not found: ID does not exist" containerID="d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7"
Sep 11 18:06:59.334627 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.334544    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7"} err="failed to get container status \"d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7\": rpc error: code = NotFound desc = could not find container \"d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7\": container with ID starting with d1839fd661d435acb966ae649aa88d1d40290d12e125375ba900b600b74c90a7 not found: ID does not exist"
Sep 11 18:06:59.363077 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.363045    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-767c56875-kd8wx"]
Sep 11 18:06:59.374283 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:06:59.374247    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-767c56875-kd8wx"]
Sep 11 18:07:00.731328 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:00.731282    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46a7d14b-6c29-4cfd-88b2-0f9ac930906f" path="/var/lib/kubelet/pods/46a7d14b-6c29-4cfd-88b2-0f9ac930906f/volumes"
Sep 11 18:07:16.130615 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.130578    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"]
Sep 11 18:07:16.131178 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.130999    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a7d14b-6c29-4cfd-88b2-0f9ac930906f" containerName="authorino"
Sep 11 18:07:16.131178 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.131017    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a7d14b-6c29-4cfd-88b2-0f9ac930906f" containerName="authorino"
Sep 11 18:07:16.131178 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.131075    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="46a7d14b-6c29-4cfd-88b2-0f9ac930906f" containerName="authorino"
Sep 11 18:07:16.135500 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.135482    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.141930 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.141736    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-93598a-openshift-default-dockercfg-kjqvn\""
Sep 11 18:07:16.164666 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.164642    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"]
Sep 11 18:07:16.285746 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285712    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjrv\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-kube-api-access-cpjrv\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.285746 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-credential-socket\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.285944 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-envoy\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.285944 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-data\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.285944 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285852    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f29dfd-097e-4f99-afe8-c004429331af-istio-podinfo\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.285944 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285894    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f29dfd-097e-4f99-afe8-c004429331af-istiod-ca-cert\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.285944 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285914    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-istio-token\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.286117 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285948    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-socket\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.286117 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.285979    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-certs\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.292659 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.292631    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"]
Sep 11 18:07:16.296116 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.296098    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.299030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.299011    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4e0b0b-openshift-default-dockercfg-kpklk\""
Sep 11 18:07:16.317276 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.317255    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"]
Sep 11 18:07:16.386736 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.386660    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjrv\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-kube-api-access-cpjrv\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.386736 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.386710    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-credential-socket\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.386955 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.386742    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-envoy\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.386955 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.386872    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-data\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.386955 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.386929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f29dfd-097e-4f99-afe8-c004429331af-istio-podinfo\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.386981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f29dfd-097e-4f99-afe8-c004429331af-istiod-ca-cert\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.387018    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-istio-token\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.387068    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-socket\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.387101    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-certs\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387338 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.387134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-credential-socket\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387790 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.387734    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f29dfd-097e-4f99-afe8-c004429331af-istiod-ca-cert\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387790 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.387760    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-data\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387980 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.387796    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-certs\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.387980 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.387824    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-socket\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.389400 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.389374    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-envoy\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.389578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.389550    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f29dfd-097e-4f99-afe8-c004429331af-istio-podinfo\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.414588 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.414563    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjrv\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-kube-api-access-cpjrv\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.424100 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.424078    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-istio-token\") pod \"e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") " pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.445101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.445074    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:16.487938 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.487893    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-podinfo\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.488162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.487969    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-data\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.488162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.488002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-envoy\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.488162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.488047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc14be55-d6a6-4ec1-8c65-9057f026741a-istiod-ca-cert\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.488162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.488103    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-socket\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.488162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.488122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-credential-socket\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.488162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.488139    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-certs\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.488162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.488157    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-token\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.488554 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.488188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnj2\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-kube-api-access-7dnj2\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589374 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589333    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-podinfo\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589540 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589389    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-data\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589540 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-envoy\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589540 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc14be55-d6a6-4ec1-8c65-9057f026741a-istiod-ca-cert\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589540 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-socket\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589540 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589510    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-credential-socket\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589802 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-certs\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589802 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589570    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-token\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589802 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnj2\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-kube-api-access-7dnj2\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589959 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589863    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-data\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.589959 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.589905    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-socket\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.590102 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.590071    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-credential-socket\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.590351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.590293    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc14be55-d6a6-4ec1-8c65-9057f026741a-istiod-ca-cert\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.590558 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.590523    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-certs\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.592421 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.592402    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-podinfo\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.592513 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.592434    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-envoy\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.600551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.600525    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-token\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.601070 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.601049    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnj2\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-kube-api-access-7dnj2\") pod \"e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") " pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.605886 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.605871    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:16.735059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.735031    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"]
Sep 11 18:07:16.735172 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:16.735034    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f29dfd_097e_4f99_afe8_c004429331af.slice/crio-09c296bf8695fb8826f45dc9f98ad7475bf82b5a98e5044a57afd9be9a114187 WatchSource:0}: Error finding container 09c296bf8695fb8826f45dc9f98ad7475bf82b5a98e5044a57afd9be9a114187: Status 404 returned error can't find the container with id 09c296bf8695fb8826f45dc9f98ad7475bf82b5a98e5044a57afd9be9a114187
Sep 11 18:07:16.737057 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.737023    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:16.737140 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.737087    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:16.737140 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.737116    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:16.895805 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.895778    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"]
Sep 11 18:07:16.896487 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:16.896460    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc14be55_d6a6_4ec1_8c65_9057f026741a.slice/crio-dd160d8e3fb715d61ce0ae9a021e1a81b0e65d7a833a691a9d95e5440928e404 WatchSource:0}: Error finding container dd160d8e3fb715d61ce0ae9a021e1a81b0e65d7a833a691a9d95e5440928e404: Status 404 returned error can't find the container with id dd160d8e3fb715d61ce0ae9a021e1a81b0e65d7a833a691a9d95e5440928e404
Sep 11 18:07:16.898887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.898845    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:16.898962 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.898921    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:16.898962 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:16.898950    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:17.421828 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.421787    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" event={"ID":"dc14be55-d6a6-4ec1-8c65-9057f026741a","Type":"ContainerStarted","Data":"2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0"}
Sep 11 18:07:17.422276 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.421887    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" event={"ID":"dc14be55-d6a6-4ec1-8c65-9057f026741a","Type":"ContainerStarted","Data":"dd160d8e3fb715d61ce0ae9a021e1a81b0e65d7a833a691a9d95e5440928e404"}
Sep 11 18:07:17.423394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.423365    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" event={"ID":"69f29dfd-097e-4f99-afe8-c004429331af","Type":"ContainerStarted","Data":"d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6"}
Sep 11 18:07:17.423482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.423420    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" event={"ID":"69f29dfd-097e-4f99-afe8-c004429331af","Type":"ContainerStarted","Data":"09c296bf8695fb8826f45dc9f98ad7475bf82b5a98e5044a57afd9be9a114187"}
Sep 11 18:07:17.445809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.445784    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:17.451421 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.451400    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:17.502215 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.502165    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" podStartSLOduration=1.502150981 podStartE2EDuration="1.502150981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:17.500931701 +0000 UTC m=+801.270998049" watchObservedRunningTime="2026-09-11 18:07:17.502150981 +0000 UTC m=+801.272217328"
Sep 11 18:07:17.545476 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.545428    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" podStartSLOduration=1.545413924 podStartE2EDuration="1.545413924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:17.544090102 +0000 UTC m=+801.314156475" watchObservedRunningTime="2026-09-11 18:07:17.545413924 +0000 UTC m=+801.315480272"
Sep 11 18:07:17.606223 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.606194    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:17.610906 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.610878    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:17.797410 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.797334    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7508-d7c0ba"]
Sep 11 18:07:17.800806 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.800789    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:17.803900 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.803878    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q6zqg\""
Sep 11 18:07:17.812874 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.812852    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7508-d7c0ba"]
Sep 11 18:07:17.902643 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:17.902613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlx8l\" (UniqueName: \"kubernetes.io/projected/df29c7c6-a57e-44c3-b554-a478b612bb5e-kube-api-access-dlx8l\") pod \"test-curl-7508-d7c0ba\" (UID: \"df29c7c6-a57e-44c3-b554-a478b612bb5e\") " pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:18.003156 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.003122    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlx8l\" (UniqueName: \"kubernetes.io/projected/df29c7c6-a57e-44c3-b554-a478b612bb5e-kube-api-access-dlx8l\") pod \"test-curl-7508-d7c0ba\" (UID: \"df29c7c6-a57e-44c3-b554-a478b612bb5e\") " pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:18.013595 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.013564    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlx8l\" (UniqueName: \"kubernetes.io/projected/df29c7c6-a57e-44c3-b554-a478b612bb5e-kube-api-access-dlx8l\") pod \"test-curl-7508-d7c0ba\" (UID: \"df29c7c6-a57e-44c3-b554-a478b612bb5e\") " pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:18.110502 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.110412    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:18.396151 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.396115    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7508-d7c0ba"]
Sep 11 18:07:18.397118 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:18.397089    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf29c7c6_a57e_44c3_b554_a478b612bb5e.slice/crio-2b04961a477adf64d3859b91920103fe9ca3bd1db90ef8bda86f24a1cd4d6eab WatchSource:0}: Error finding container 2b04961a477adf64d3859b91920103fe9ca3bd1db90ef8bda86f24a1cd4d6eab: Status 404 returned error can't find the container with id 2b04961a477adf64d3859b91920103fe9ca3bd1db90ef8bda86f24a1cd4d6eab
Sep 11 18:07:18.429794 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.429764    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7508-d7c0ba" event={"ID":"df29c7c6-a57e-44c3-b554-a478b612bb5e","Type":"ContainerStarted","Data":"2b04961a477adf64d3859b91920103fe9ca3bd1db90ef8bda86f24a1cd4d6eab"}
Sep 11 18:07:18.430118 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.430076    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:18.430167 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.430128    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:18.430834 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.430818    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:18.430943 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:18.430928    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:19.443369 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:19.443329    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7508-d7c0ba" event={"ID":"df29c7c6-a57e-44c3-b554-a478b612bb5e","Type":"ContainerStarted","Data":"54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781"}
Sep 11 18:07:19.477929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:19.477866    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7508-d7c0ba" podStartSLOduration=1.965458419 podStartE2EDuration="2.477844473s" totalImagesPullingTime="512.386054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:19.475481873 +0000 UTC m=+803.245548221" watchObservedRunningTime="2026-09-11 18:07:19.477844473 +0000 UTC m=+803.247910823"
Sep 11 18:07:20.335426 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.335335    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7508-d7c0ba"]
Sep 11 18:07:20.348054 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.348024    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7508-d7c0ba"]
Sep 11 18:07:20.455792 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:07:20.455760    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7508-d7c0ba" container="test-curl-7508-d7c0ba"
Sep 11 18:07:20.495529 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.495487    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7508-c9c54e"]
Sep 11 18:07:20.495878 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.495863    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df29c7c6-a57e-44c3-b554-a478b612bb5e" containerName="test-curl-7508-d7c0ba"
Sep 11 18:07:20.495878 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.495877    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="df29c7c6-a57e-44c3-b554-a478b612bb5e" containerName="test-curl-7508-d7c0ba"
Sep 11 18:07:20.496035 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.495949    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="df29c7c6-a57e-44c3-b554-a478b612bb5e" containerName="test-curl-7508-d7c0ba"
Sep 11 18:07:20.499176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.499154    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:20.510802 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.510776    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7508-c9c54e"]
Sep 11 18:07:20.522625 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.522584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxs7f\" (UniqueName: \"kubernetes.io/projected/e52f7729-f46f-4b11-9fd2-8afe0a7fd33b-kube-api-access-zxs7f\") pod \"test-curl-7508-c9c54e\" (UID: \"e52f7729-f46f-4b11-9fd2-8afe0a7fd33b\") " pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:20.623482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.623398    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxs7f\" (UniqueName: \"kubernetes.io/projected/e52f7729-f46f-4b11-9fd2-8afe0a7fd33b-kube-api-access-zxs7f\") pod \"test-curl-7508-c9c54e\" (UID: \"e52f7729-f46f-4b11-9fd2-8afe0a7fd33b\") " pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:20.637676 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.637646    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxs7f\" (UniqueName: \"kubernetes.io/projected/e52f7729-f46f-4b11-9fd2-8afe0a7fd33b-kube-api-access-zxs7f\") pod \"test-curl-7508-c9c54e\" (UID: \"e52f7729-f46f-4b11-9fd2-8afe0a7fd33b\") " pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:20.809364 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:20.809323    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:21.103388 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:21.103356    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7508-c9c54e"]
Sep 11 18:07:21.104606 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:21.104583    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode52f7729_f46f_4b11_9fd2_8afe0a7fd33b.slice/crio-825949de8c2717c0c18fea5655be364785f1afe25218e8353feb5879fa9e8b4e WatchSource:0}: Error finding container 825949de8c2717c0c18fea5655be364785f1afe25218e8353feb5879fa9e8b4e: Status 404 returned error can't find the container with id 825949de8c2717c0c18fea5655be364785f1afe25218e8353feb5879fa9e8b4e
Sep 11 18:07:21.464600 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:21.464567    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7508-c9c54e" event={"ID":"e52f7729-f46f-4b11-9fd2-8afe0a7fd33b","Type":"ContainerStarted","Data":"fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b"}
Sep 11 18:07:21.465037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:21.464609    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7508-c9c54e" event={"ID":"e52f7729-f46f-4b11-9fd2-8afe0a7fd33b","Type":"ContainerStarted","Data":"825949de8c2717c0c18fea5655be364785f1afe25218e8353feb5879fa9e8b4e"}
Sep 11 18:07:21.465037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:21.464702    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7508-d7c0ba" podUID="df29c7c6-a57e-44c3-b554-a478b612bb5e" containerName="test-curl-7508-d7c0ba" containerID="cri-o://54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781" gracePeriod=2
Sep 11 18:07:21.491597 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:21.491550    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7508-c9c54e" podStartSLOduration=1.491537137 podStartE2EDuration="1.491537137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:21.488078543 +0000 UTC m=+805.258144891" watchObservedRunningTime="2026-09-11 18:07:21.491537137 +0000 UTC m=+805.261603485"
Sep 11 18:07:22.301324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.301271    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7508-c9c54e"]
Sep 11 18:07:22.316903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.316830    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7508-c9c54e"]
Sep 11 18:07:22.470520 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:07:22.470492    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7508-c9c54e" container="test-curl-7508-c9c54e"
Sep 11 18:07:22.532289 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.532257    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"]
Sep 11 18:07:22.532751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.532731    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" containerName="test-curl-7508-c9c54e"
Sep 11 18:07:22.532871 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.532754    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" containerName="test-curl-7508-c9c54e"
Sep 11 18:07:22.532871 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.532849    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" containerName="test-curl-7508-c9c54e"
Sep 11 18:07:22.535948 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.535931    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:22.537776 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.537752    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wpq\" (UniqueName: \"kubernetes.io/projected/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-kube-api-access-d4wpq\") pod \"payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn\" (UID: \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:22.537872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.537802    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn\" (UID: \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:22.538799 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.538778    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f8a2d4-dockercfg-9pl2h\""
Sep 11 18:07:22.538879 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.538780    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f8a2d4\""
Sep 11 18:07:22.549285 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.549264    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"]
Sep 11 18:07:22.563210 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.563187    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"]
Sep 11 18:07:22.567057 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.567007    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:22.580104 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.580081    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"]
Sep 11 18:07:22.638444 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.638412    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch\" (UID: \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:22.638610 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.638477    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wpq\" (UniqueName: \"kubernetes.io/projected/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-kube-api-access-d4wpq\") pod \"payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn\" (UID: \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:22.638610 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.638504    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh5mn\" (UniqueName: \"kubernetes.io/projected/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-kube-api-access-sh5mn\") pod \"payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch\" (UID: \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:22.638610 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.638550    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn\" (UID: \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:22.639204 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.639186    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn\" (UID: \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:22.651730 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.651703    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wpq\" (UniqueName: \"kubernetes.io/projected/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-kube-api-access-d4wpq\") pod \"payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn\" (UID: \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:22.739787 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.739757    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh5mn\" (UniqueName: \"kubernetes.io/projected/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-kube-api-access-sh5mn\") pod \"payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch\" (UID: \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:22.739954 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.739820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch\" (UID: \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:22.740526 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.740508    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch\" (UID: \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:22.753366 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.753337    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh5mn\" (UniqueName: \"kubernetes.io/projected/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-kube-api-access-sh5mn\") pod \"payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch\" (UID: \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:22.846383 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.846280    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:22.878604 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:22.878574    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:23.129953 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:23.126893    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode045d9d6_fe59_48a2_9eb6_b46af76cbf00.slice/crio-929b37ac6a6fe05dd0e7b61db739126bb05c8376fea87bad4e0875a43a20937c WatchSource:0}: Error finding container 929b37ac6a6fe05dd0e7b61db739126bb05c8376fea87bad4e0875a43a20937c: Status 404 returned error can't find the container with id 929b37ac6a6fe05dd0e7b61db739126bb05c8376fea87bad4e0875a43a20937c
Sep 11 18:07:23.129953 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.127518    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"]
Sep 11 18:07:23.180337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.180316    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"]
Sep 11 18:07:23.181837 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:23.181813    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2f9d5d_1a9b_4c79_8208_77f0a0fa4bc1.slice/crio-6306381d93011ef05ebe5b49882a784aed408bace0dd75729150958e0e14becf WatchSource:0}: Error finding container 6306381d93011ef05ebe5b49882a784aed408bace0dd75729150958e0e14becf: Status 404 returned error can't find the container with id 6306381d93011ef05ebe5b49882a784aed408bace0dd75729150958e0e14becf
Sep 11 18:07:23.198227 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.198206    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"]
Sep 11 18:07:23.202212 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.202194    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.205917 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.205899    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-736ba3-openshift-default-dockercfg-zrm64\""
Sep 11 18:07:23.220613 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.220590    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"]
Sep 11 18:07:23.244252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244223    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-envoy\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.244398 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244263    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp8f6\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-kube-api-access-pp8f6\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.244398 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244322    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-podinfo\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.244398 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244375    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-token\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.244566 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244412    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e966a2f4-4fcc-4478-93b8-13100af36dd8-istiod-ca-cert\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.244566 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-credential-socket\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.244751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-socket\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.244751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-certs\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.244751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.244664    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-data\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345509 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-socket\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-certs\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345562    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-data\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345597    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-envoy\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345626    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp8f6\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-kube-api-access-pp8f6\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345656    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-podinfo\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345839 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345686    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-token\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345839 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345715    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e966a2f4-4fcc-4478-93b8-13100af36dd8-istiod-ca-cert\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345839 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345754    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-credential-socket\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.345971 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345958    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-socket\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.346018 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.345992    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-certs\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.346055 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.346027    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-credential-socket\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.346549 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.346520    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-data\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.346798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.346772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e966a2f4-4fcc-4478-93b8-13100af36dd8-istiod-ca-cert\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.348899 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.348855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-envoy\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.349186 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.349163    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-podinfo\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.360235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.360180    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-token\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.361228 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.361208    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp8f6\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-kube-api-access-pp8f6\") pod \"e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") " pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.479447 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.479412    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch" event={"ID":"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1","Type":"ContainerStarted","Data":"fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82"}
Sep 11 18:07:23.479836 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.479475    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch" event={"ID":"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1","Type":"ContainerStarted","Data":"6306381d93011ef05ebe5b49882a784aed408bace0dd75729150958e0e14becf"}
Sep 11 18:07:23.479836 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.479528    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:23.481610 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.481588    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn" event={"ID":"e045d9d6-fe59-48a2-9eb6-b46af76cbf00","Type":"ContainerStarted","Data":"6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac"}
Sep 11 18:07:23.481717 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.481626    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn" event={"ID":"e045d9d6-fe59-48a2-9eb6-b46af76cbf00","Type":"ContainerStarted","Data":"929b37ac6a6fe05dd0e7b61db739126bb05c8376fea87bad4e0875a43a20937c"}
Sep 11 18:07:23.481717 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.481675    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:23.481830 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.481742    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7508-c9c54e" podUID="e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" containerName="test-curl-7508-c9c54e" containerID="cri-o://fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b" gracePeriod=2
Sep 11 18:07:23.506688 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.506646    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch" podStartSLOduration=1.506629824 podStartE2EDuration="1.506629824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:23.503968698 +0000 UTC m=+807.274035050" watchObservedRunningTime="2026-09-11 18:07:23.506629824 +0000 UTC m=+807.276696163"
Sep 11 18:07:23.524958 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.524929    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:23.525756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.525715    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn" podStartSLOduration=1.525697222 podStartE2EDuration="1.525697222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:23.524419517 +0000 UTC m=+807.294485866" watchObservedRunningTime="2026-09-11 18:07:23.525697222 +0000 UTC m=+807.295763571"
Sep 11 18:07:23.699192 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.699170    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:23.755193 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.755112    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlx8l\" (UniqueName: \"kubernetes.io/projected/df29c7c6-a57e-44c3-b554-a478b612bb5e-kube-api-access-dlx8l\") pod \"df29c7c6-a57e-44c3-b554-a478b612bb5e\" (UID: \"df29c7c6-a57e-44c3-b554-a478b612bb5e\") "
Sep 11 18:07:23.757725 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.757696    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df29c7c6-a57e-44c3-b554-a478b612bb5e-kube-api-access-dlx8l" (OuterVolumeSpecName: "kube-api-access-dlx8l") pod "df29c7c6-a57e-44c3-b554-a478b612bb5e" (UID: "df29c7c6-a57e-44c3-b554-a478b612bb5e"). InnerVolumeSpecName "kube-api-access-dlx8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:07:23.827960 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.827932    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"]
Sep 11 18:07:23.828384 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:23.828357    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode966a2f4_4fcc_4478_93b8_13100af36dd8.slice/crio-c6c04f92a041bd17996fe5ead0be9e40b3dffac5a4cb0952b7f9e82aa9d4a736 WatchSource:0}: Error finding container c6c04f92a041bd17996fe5ead0be9e40b3dffac5a4cb0952b7f9e82aa9d4a736: Status 404 returned error can't find the container with id c6c04f92a041bd17996fe5ead0be9e40b3dffac5a4cb0952b7f9e82aa9d4a736
Sep 11 18:07:23.830603 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.830573    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:23.830682 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.830642    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:23.830739 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.830690    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:23.856463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:23.856441    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlx8l\" (UniqueName: \"kubernetes.io/projected/df29c7c6-a57e-44c3-b554-a478b612bb5e-kube-api-access-dlx8l\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:07:24.489757 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.489716    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" event={"ID":"e966a2f4-4fcc-4478-93b8-13100af36dd8","Type":"ContainerStarted","Data":"9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7"}
Sep 11 18:07:24.490200 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.489817    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" event={"ID":"e966a2f4-4fcc-4478-93b8-13100af36dd8","Type":"ContainerStarted","Data":"c6c04f92a041bd17996fe5ead0be9e40b3dffac5a4cb0952b7f9e82aa9d4a736"}
Sep 11 18:07:24.491050 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.491021    2571 generic.go:356] "Generic (PLEG): container finished" podID="df29c7c6-a57e-44c3-b554-a478b612bb5e" containerID="54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781" exitCode=137
Sep 11 18:07:24.491141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.491072    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:24.491141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.491116    2571 scope.go:117] "RemoveContainer" containerID="54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781"
Sep 11 18:07:24.500989 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.500964    2571 scope.go:117] "RemoveContainer" containerID="54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781"
Sep 11 18:07:24.501239 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:07:24.501222    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781\": container with ID starting with 54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781 not found: ID does not exist" containerID="54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781"
Sep 11 18:07:24.501354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.501246    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781"} err="failed to get container status \"54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781\": rpc error: code = NotFound desc = could not find container \"54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781\": container with ID starting with 54a893730a81d60ef491662c9e9f02e347e6a2aade718589139d271706061781 not found: ID does not exist"
Sep 11 18:07:24.523141 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:07:24.523107    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7508-d7c0ba\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="df29c7c6-a57e-44c3-b554-a478b612bb5e" pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:24.523588 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.523556    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" podStartSLOduration=1.5235444 podStartE2EDuration="1.5235444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:24.52067275 +0000 UTC m=+808.290739098" watchObservedRunningTime="2026-09-11 18:07:24.5235444 +0000 UTC m=+808.293610747"
Sep 11 18:07:24.525189 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.525169    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:24.530142 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.530125    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:24.532955 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:07:24.532936    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7508-d7c0ba\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="df29c7c6-a57e-44c3-b554-a478b612bb5e" pod="opendatahub/test-curl-7508-d7c0ba"
Sep 11 18:07:24.730937 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:24.730901    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df29c7c6-a57e-44c3-b554-a478b612bb5e" path="/var/lib/kubelet/pods/df29c7c6-a57e-44c3-b554-a478b612bb5e/volumes"
Sep 11 18:07:25.232713 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.232627    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"]
Sep 11 18:07:25.237624 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.237606    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"]
Sep 11 18:07:25.237731 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.237710    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:25.240510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.240484    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-93598a\""
Sep 11 18:07:25.241821 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.241806    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:25.249926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.249903    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"]
Sep 11 18:07:25.253710 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.253688    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"]
Sep 11 18:07:25.269239 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.269217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9add17f0-91fc-48ff-86ef-7f025016b04e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx\" (UID: \"9add17f0-91fc-48ff-86ef-7f025016b04e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:25.269353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.269245    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpt2h\" (UniqueName: \"kubernetes.io/projected/57dc7d24-87a4-41f0-a5bf-8afc71170e54-kube-api-access-kpt2h\") pod \"payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz\" (UID: \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:25.269353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.269267    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dc7d24-87a4-41f0-a5bf-8afc71170e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz\" (UID: \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:25.269353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.269284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5tjc\" (UniqueName: \"kubernetes.io/projected/9add17f0-91fc-48ff-86ef-7f025016b04e-kube-api-access-n5tjc\") pod \"payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx\" (UID: \"9add17f0-91fc-48ff-86ef-7f025016b04e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:25.370582 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.370552    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9add17f0-91fc-48ff-86ef-7f025016b04e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx\" (UID: \"9add17f0-91fc-48ff-86ef-7f025016b04e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:25.370775 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.370589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpt2h\" (UniqueName: \"kubernetes.io/projected/57dc7d24-87a4-41f0-a5bf-8afc71170e54-kube-api-access-kpt2h\") pod \"payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz\" (UID: \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:25.370775 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.370624    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dc7d24-87a4-41f0-a5bf-8afc71170e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz\" (UID: \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:25.370775 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.370652    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5tjc\" (UniqueName: \"kubernetes.io/projected/9add17f0-91fc-48ff-86ef-7f025016b04e-kube-api-access-n5tjc\") pod \"payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx\" (UID: \"9add17f0-91fc-48ff-86ef-7f025016b04e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:25.371405 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.371367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9add17f0-91fc-48ff-86ef-7f025016b04e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx\" (UID: \"9add17f0-91fc-48ff-86ef-7f025016b04e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:25.371594 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.371565    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dc7d24-87a4-41f0-a5bf-8afc71170e54-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz\" (UID: \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:25.387018 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.383840    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpt2h\" (UniqueName: \"kubernetes.io/projected/57dc7d24-87a4-41f0-a5bf-8afc71170e54-kube-api-access-kpt2h\") pod \"payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz\" (UID: \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:25.387018 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.383848    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5tjc\" (UniqueName: \"kubernetes.io/projected/9add17f0-91fc-48ff-86ef-7f025016b04e-kube-api-access-n5tjc\") pod \"payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx\" (UID: \"9add17f0-91fc-48ff-86ef-7f025016b04e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:25.499665 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.499639    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:25.500515 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.500493    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:25.555229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.555199    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:25.563951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.563924    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:25.670216 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.670189    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:25.774637 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.774558    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxs7f\" (UniqueName: \"kubernetes.io/projected/e52f7729-f46f-4b11-9fd2-8afe0a7fd33b-kube-api-access-zxs7f\") pod \"e52f7729-f46f-4b11-9fd2-8afe0a7fd33b\" (UID: \"e52f7729-f46f-4b11-9fd2-8afe0a7fd33b\") "
Sep 11 18:07:25.776643 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.776617    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e52f7729-f46f-4b11-9fd2-8afe0a7fd33b-kube-api-access-zxs7f" (OuterVolumeSpecName: "kube-api-access-zxs7f") pod "e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" (UID: "e52f7729-f46f-4b11-9fd2-8afe0a7fd33b"). InnerVolumeSpecName "kube-api-access-zxs7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:07:25.853553 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:25.853519    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9add17f0_91fc_48ff_86ef_7f025016b04e.slice/crio-3255267e2987a8eda1c15626df4ff340dd73a0c6da58b6f49ba4e5d2493bff8d WatchSource:0}: Error finding container 3255267e2987a8eda1c15626df4ff340dd73a0c6da58b6f49ba4e5d2493bff8d: Status 404 returned error can't find the container with id 3255267e2987a8eda1c15626df4ff340dd73a0c6da58b6f49ba4e5d2493bff8d
Sep 11 18:07:25.854562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.854533    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"]
Sep 11 18:07:25.856661 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:25.856383    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dc7d24_87a4_41f0_a5bf_8afc71170e54.slice/crio-e33d6a837458e770d3cda5d130c500610ffd535c8cd1c90545e8af767f4386b0 WatchSource:0}: Error finding container e33d6a837458e770d3cda5d130c500610ffd535c8cd1c90545e8af767f4386b0: Status 404 returned error can't find the container with id e33d6a837458e770d3cda5d130c500610ffd535c8cd1c90545e8af767f4386b0
Sep 11 18:07:25.858190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.858149    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"]
Sep 11 18:07:25.875837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:25.875812    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxs7f\" (UniqueName: \"kubernetes.io/projected/e52f7729-f46f-4b11-9fd2-8afe0a7fd33b-kube-api-access-zxs7f\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:07:26.506511 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.506478    2571 generic.go:356] "Generic (PLEG): container finished" podID="e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" containerID="fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b" exitCode=137
Sep 11 18:07:26.506957 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.506543    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:26.506957 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.506580    2571 scope.go:117] "RemoveContainer" containerID="fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b"
Sep 11 18:07:26.508611 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.508579    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz" event={"ID":"57dc7d24-87a4-41f0-a5bf-8afc71170e54","Type":"ContainerStarted","Data":"1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa"}
Sep 11 18:07:26.508743 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.508624    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz" event={"ID":"57dc7d24-87a4-41f0-a5bf-8afc71170e54","Type":"ContainerStarted","Data":"e33d6a837458e770d3cda5d130c500610ffd535c8cd1c90545e8af767f4386b0"}
Sep 11 18:07:26.508743 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.508653    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:26.510672 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.510648    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx" event={"ID":"9add17f0-91fc-48ff-86ef-7f025016b04e","Type":"ContainerStarted","Data":"e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac"}
Sep 11 18:07:26.510773 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.510693    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx" event={"ID":"9add17f0-91fc-48ff-86ef-7f025016b04e","Type":"ContainerStarted","Data":"3255267e2987a8eda1c15626df4ff340dd73a0c6da58b6f49ba4e5d2493bff8d"}
Sep 11 18:07:26.510860 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.510846    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:26.517160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.517145    2571 scope.go:117] "RemoveContainer" containerID="fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b"
Sep 11 18:07:26.517441 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:07:26.517420    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b\": container with ID starting with fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b not found: ID does not exist" containerID="fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b"
Sep 11 18:07:26.517504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.517444    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b"} err="failed to get container status \"fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b\": rpc error: code = NotFound desc = could not find container \"fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b\": container with ID starting with fe2ec5c6380aed5e0e9ca6f42a35a743ab7cc8fb69bd308f8b372177f93afc4b not found: ID does not exist"
Sep 11 18:07:26.536512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.536470    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz" podStartSLOduration=1.53645683 podStartE2EDuration="1.53645683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:26.534094832 +0000 UTC m=+810.304161181" watchObservedRunningTime="2026-09-11 18:07:26.53645683 +0000 UTC m=+810.306523178"
Sep 11 18:07:26.537151 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:07:26.537122    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7508-c9c54e\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:26.568848 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.568801    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx" podStartSLOduration=1.5687861459999999 podStartE2EDuration="1.568786146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:26.568667658 +0000 UTC m=+810.338734006" watchObservedRunningTime="2026-09-11 18:07:26.568786146 +0000 UTC m=+810.338852494"
Sep 11 18:07:26.738063 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:07:26.738013    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7508-c9c54e\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" pod="opendatahub/test-curl-7508-c9c54e"
Sep 11 18:07:26.741730 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:26.741698    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e52f7729-f46f-4b11-9fd2-8afe0a7fd33b" path="/var/lib/kubelet/pods/e52f7729-f46f-4b11-9fd2-8afe0a7fd33b/volumes"
Sep 11 18:07:30.479490 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.479451    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"]
Sep 11 18:07:30.484625 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.484603    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:30.487814 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.487795    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-736ba3\""
Sep 11 18:07:30.487937 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.487920    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-736ba3-dockercfg-pvcvv\""
Sep 11 18:07:30.498650 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.498622    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"]
Sep 11 18:07:30.522658 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.522634    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62556da9-496e-42aa-85f2-1744fc2791e2-plugins-config-volume\") pod \"payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l\" (UID: \"62556da9-496e-42aa-85f2-1744fc2791e2\") " pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:30.522872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.522852    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf7rb\" (UniqueName: \"kubernetes.io/projected/62556da9-496e-42aa-85f2-1744fc2791e2-kube-api-access-rf7rb\") pod \"payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l\" (UID: \"62556da9-496e-42aa-85f2-1744fc2791e2\") " pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:30.623711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.623673    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf7rb\" (UniqueName: \"kubernetes.io/projected/62556da9-496e-42aa-85f2-1744fc2791e2-kube-api-access-rf7rb\") pod \"payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l\" (UID: \"62556da9-496e-42aa-85f2-1744fc2791e2\") " pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:30.623856 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.623805    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62556da9-496e-42aa-85f2-1744fc2791e2-plugins-config-volume\") pod \"payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l\" (UID: \"62556da9-496e-42aa-85f2-1744fc2791e2\") " pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:30.624594 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.624574    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62556da9-496e-42aa-85f2-1744fc2791e2-plugins-config-volume\") pod \"payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l\" (UID: \"62556da9-496e-42aa-85f2-1744fc2791e2\") " pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:30.643607 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.643573    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf7rb\" (UniqueName: \"kubernetes.io/projected/62556da9-496e-42aa-85f2-1744fc2791e2-kube-api-access-rf7rb\") pod \"payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l\" (UID: \"62556da9-496e-42aa-85f2-1744fc2791e2\") " pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:30.795142 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.795067    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:30.889590 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.889555    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"]
Sep 11 18:07:30.895366 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.895314    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.898995 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.898974    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-33958e97-openshift-default-dockercfg-ft4vk\""
Sep 11 18:07:30.901614 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.901592    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"]
Sep 11 18:07:30.927289 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.927260    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlqgs\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-kube-api-access-hlqgs\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.927533 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.927513    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-socket\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.927663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.927650    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41ba5e4-c105-4400-8cbb-90087a386a2a-istiod-ca-cert\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.927796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.927768    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-podinfo\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.927912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.927899    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-certs\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.928017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.928002    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-token\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.928141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.928128    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-envoy\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.928258 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.928246    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-credential-socket\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:30.928437 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:30.928422    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-data\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029267 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029229    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-credential-socket\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029289    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-data\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlqgs\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-kube-api-access-hlqgs\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-socket\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029370    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41ba5e4-c105-4400-8cbb-90087a386a2a-istiod-ca-cert\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-podinfo\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-certs\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029425    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-token\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029482 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029454    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-envoy\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029932 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029623    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-data\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.029932 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029696    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-credential-socket\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.030044 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.029955    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-certs\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.030101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.030061    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41ba5e4-c105-4400-8cbb-90087a386a2a-istiod-ca-cert\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.030162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.030113    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-socket\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.031516 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.031497    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-envoy\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.032214 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.032192    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-podinfo\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.045904 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.045844    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlqgs\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-kube-api-access-hlqgs\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.045904 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.045884    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-token\") pod \"e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") " pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.081559 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.081531    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"]
Sep 11 18:07:31.082371 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:31.082342    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62556da9_496e_42aa_85f2_1744fc2791e2.slice/crio-7339591fb264229b5887cd5c48d5d99c8dcb9ef0e3a94a4fae87fcfbe19d7b8f WatchSource:0}: Error finding container 7339591fb264229b5887cd5c48d5d99c8dcb9ef0e3a94a4fae87fcfbe19d7b8f: Status 404 returned error can't find the container with id 7339591fb264229b5887cd5c48d5d99c8dcb9ef0e3a94a4fae87fcfbe19d7b8f
Sep 11 18:07:31.209073 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.209031    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:31.529136 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.529110    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"]
Sep 11 18:07:31.530461 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:31.530437    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41ba5e4_c105_4400_8cbb_90087a386a2a.slice/crio-781a85092549d882eab87245b4f61a55cef253f051d375993da3ce54a9c783bf WatchSource:0}: Error finding container 781a85092549d882eab87245b4f61a55cef253f051d375993da3ce54a9c783bf: Status 404 returned error can't find the container with id 781a85092549d882eab87245b4f61a55cef253f051d375993da3ce54a9c783bf
Sep 11 18:07:31.532550 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.532520    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:31.532630 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.532583    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:31.532630 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.532615    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:31.544125 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.544102    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l" event={"ID":"62556da9-496e-42aa-85f2-1744fc2791e2","Type":"ContainerStarted","Data":"2fb65d4281108edfbe9c9b40677726faa1ee81d9345ba8f7df31f080e52fedc8"}
Sep 11 18:07:31.544213 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.544154    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l" event={"ID":"62556da9-496e-42aa-85f2-1744fc2791e2","Type":"ContainerStarted","Data":"7339591fb264229b5887cd5c48d5d99c8dcb9ef0e3a94a4fae87fcfbe19d7b8f"}
Sep 11 18:07:31.544213 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.544194    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:31.545479 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:31.545460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" event={"ID":"b41ba5e4-c105-4400-8cbb-90087a386a2a","Type":"ContainerStarted","Data":"781a85092549d882eab87245b4f61a55cef253f051d375993da3ce54a9c783bf"}
Sep 11 18:07:32.557204 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:32.557162    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" event={"ID":"b41ba5e4-c105-4400-8cbb-90087a386a2a","Type":"ContainerStarted","Data":"2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f"}
Sep 11 18:07:32.583469 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:32.583421    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l" podStartSLOduration=2.583404157 podStartE2EDuration="2.583404157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:31.573057647 +0000 UTC m=+815.343123996" watchObservedRunningTime="2026-09-11 18:07:32.583404157 +0000 UTC m=+816.353470505"
Sep 11 18:07:32.584885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:32.584853    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podStartSLOduration=2.584842349 podStartE2EDuration="2.584842349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:32.582599037 +0000 UTC m=+816.352665387" watchObservedRunningTime="2026-09-11 18:07:32.584842349 +0000 UTC m=+816.354908704"
Sep 11 18:07:33.209452 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:33.209416    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:33.214258 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:33.214235    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:33.565752 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:33.565660    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:33.568025 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:33.568004    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:07:34.493236 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:34.493208    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:07:34.493451 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:34.493272    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:07:37.520786 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:37.520756    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:07:37.521371 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:37.520842    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:07:37.600948 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:37.600912    2571 generic.go:356] "Generic (PLEG): container finished" podID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerID="2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0" exitCode=0
Sep 11 18:07:37.601140 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:37.600990    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" event={"ID":"dc14be55-d6a6-4ec1-8c65-9057f026741a","Type":"ContainerDied","Data":"2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0"}
Sep 11 18:07:37.601424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:37.601404    2571 scope.go:117] "RemoveContainer" containerID="2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0"
Sep 11 18:07:37.602590 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:37.602563    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:37.602672 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:37.602627    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:37.602672 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:37.602658    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:38.609629 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:38.609598    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" event={"ID":"dc14be55-d6a6-4ec1-8c65-9057f026741a","Type":"ContainerStarted","Data":"ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27"}
Sep 11 18:07:39.606680 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:39.606612    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:39.606947 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:39.606924    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 18:07:39.606996 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:39.606974    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 18:07:40.126461 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.126432    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"]
Sep 11 18:07:40.130711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.130658    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.134069 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.133897    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d59a02-openshift-default-dockercfg-4dbth\""
Sep 11 18:07:40.164324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.161503    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"]
Sep 11 18:07:40.221889 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.221854    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-socket\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.221889 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.221890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-credential-socket\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.222129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.221908    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-istio-token\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.222129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.221941    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c628866d-0242-49cd-b3ef-766ad525bfce-istio-podinfo\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.222129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.221973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-data\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.222129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.221994    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-certs\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.222129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.222034    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl9vj\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-kube-api-access-fl9vj\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.222129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.222066    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c628866d-0242-49cd-b3ef-766ad525bfce-istiod-ca-cert\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.222129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.222081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-envoy\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323088 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323053    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-socket\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323103    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-credential-socket\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-istio-token\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323177    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c628866d-0242-49cd-b3ef-766ad525bfce-istio-podinfo\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323208    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-data\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323246    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-certs\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323605 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323333    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl9vj\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-kube-api-access-fl9vj\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323605 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323381    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c628866d-0242-49cd-b3ef-766ad525bfce-istiod-ca-cert\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.323605 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.323424    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-envoy\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.324404 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.324378    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-credential-socket\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.324713 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.324691    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-socket\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.325348 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.325330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-data\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.325976 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.325926    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c628866d-0242-49cd-b3ef-766ad525bfce-istiod-ca-cert\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.326241 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.326220    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-envoy\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.326394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.326317    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-certs\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.330861 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.330799    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c628866d-0242-49cd-b3ef-766ad525bfce-istio-podinfo\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.336924 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.336873    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-istio-token\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.343920 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.343886    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl9vj\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-kube-api-access-fl9vj\") pod \"e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") " pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.443151 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.443113    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:40.610321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.609765    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 18:07:40.610321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:40.609831    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 18:07:41.029375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.028277    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"]
Sep 11 18:07:41.039816 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:41.038915    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc628866d_0242_49cd_b3ef_766ad525bfce.slice/crio-4c1452300c4964729cfdb243881afb2ac720a067f0a081514844748447afc933 WatchSource:0}: Error finding container 4c1452300c4964729cfdb243881afb2ac720a067f0a081514844748447afc933: Status 404 returned error can't find the container with id 4c1452300c4964729cfdb243881afb2ac720a067f0a081514844748447afc933
Sep 11 18:07:41.045342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.042444    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:41.045342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.042532    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:41.045342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.042599    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:41.606717 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.606679    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 18:07:41.607257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.607224    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 18:07:41.648339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.647809    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" event={"ID":"c628866d-0242-49cd-b3ef-766ad525bfce","Type":"ContainerStarted","Data":"61585775a3e7c365c7361d4eba735ac4674ad459e28ba33e0f30ec959707cda1"}
Sep 11 18:07:41.648339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.647876    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" event={"ID":"c628866d-0242-49cd-b3ef-766ad525bfce","Type":"ContainerStarted","Data":"4c1452300c4964729cfdb243881afb2ac720a067f0a081514844748447afc933"}
Sep 11 18:07:41.673162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:41.673094    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podStartSLOduration=1.6730705750000001 podStartE2EDuration="1.673070575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:41.671750942 +0000 UTC m=+825.441817291" watchObservedRunningTime="2026-09-11 18:07:41.673070575 +0000 UTC m=+825.443136924"
Sep 11 18:07:42.444530 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:42.444490    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:42.447035 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:42.447009    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 18:07:42.447444 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:42.447417    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 18:07:42.560621 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:42.560528    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:07:42.640326 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:42.637923    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 18:07:42.640326 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:42.637985    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 18:07:43.458687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:43.458648    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:43.606614 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:43.606557    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 11 18:07:43.606899 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:43.606869    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 11 18:07:43.709456 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:43.709367    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:43.714952 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:43.714924    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:07:44.611536 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:44.611498    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:44.611868 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:44.611845    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:44.612677 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:44.612657    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:07:48.350541 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.350502    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"]
Sep 11 18:07:48.357265 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.356405    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:48.367441 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.362030    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d59a02\""
Sep 11 18:07:48.371404 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.371372    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"]
Sep 11 18:07:48.500622 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.500579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf028d7-ebfa-4768-8728-96adb66e7b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq\" (UID: \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:48.500819 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.500632    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8gh\" (UniqueName: \"kubernetes.io/projected/ecf028d7-ebfa-4768-8728-96adb66e7b7d-kube-api-access-8j8gh\") pod \"payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq\" (UID: \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:48.602338 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.602240    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf028d7-ebfa-4768-8728-96adb66e7b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq\" (UID: \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:48.602338 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.602280    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8gh\" (UniqueName: \"kubernetes.io/projected/ecf028d7-ebfa-4768-8728-96adb66e7b7d-kube-api-access-8j8gh\") pod \"payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq\" (UID: \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:48.604645 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.603073    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf028d7-ebfa-4768-8728-96adb66e7b7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq\" (UID: \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:48.615017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.614880    2571 patch_prober.go:28] interesting pod/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:39278->10.132.0.51:15021: read: connection reset by peer" start-of-body=
Sep 11 18:07:48.615017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.614957    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:39278->10.132.0.51:15021: read: connection reset by peer"
Sep 11 18:07:48.618468 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.618438    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8gh\" (UniqueName: \"kubernetes.io/projected/ecf028d7-ebfa-4768-8728-96adb66e7b7d-kube-api-access-8j8gh\") pod \"payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq\" (UID: \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:48.675161 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.675132    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:48.751440 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.751385    2571 generic.go:356] "Generic (PLEG): container finished" podID="69f29dfd-097e-4f99-afe8-c004429331af" containerID="d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6" exitCode=0
Sep 11 18:07:48.751609 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.751474    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" event={"ID":"69f29dfd-097e-4f99-afe8-c004429331af","Type":"ContainerDied","Data":"d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6"}
Sep 11 18:07:48.751992 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.751971    2571 scope.go:117] "RemoveContainer" containerID="d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6"
Sep 11 18:07:48.753907 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.753868    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:48.754019 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.753961    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:48.754019 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.754006    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:48.954417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:48.954127    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"]
Sep 11 18:07:48.956715 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:07:48.956685    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf028d7_ebfa_4768_8728_96adb66e7b7d.slice/crio-7750f77cd5630849e0ef7ab541e4f8fe66ce40409b23273bcf9b02ef2a9e7d2d WatchSource:0}: Error finding container 7750f77cd5630849e0ef7ab541e4f8fe66ce40409b23273bcf9b02ef2a9e7d2d: Status 404 returned error can't find the container with id 7750f77cd5630849e0ef7ab541e4f8fe66ce40409b23273bcf9b02ef2a9e7d2d
Sep 11 18:07:49.765888 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.764915    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" event={"ID":"69f29dfd-097e-4f99-afe8-c004429331af","Type":"ContainerStarted","Data":"a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624"}
Sep 11 18:07:49.769126 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.769094    2571 generic.go:356] "Generic (PLEG): container finished" podID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerID="9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7" exitCode=0
Sep 11 18:07:49.769286 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.769152    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" event={"ID":"e966a2f4-4fcc-4478-93b8-13100af36dd8","Type":"ContainerDied","Data":"9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7"}
Sep 11 18:07:49.769747 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.769724    2571 scope.go:117] "RemoveContainer" containerID="9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7"
Sep 11 18:07:49.772048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.771577    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:49.772048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.771664    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:49.772048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.771714    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:07:49.779626 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.779586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq" event={"ID":"ecf028d7-ebfa-4768-8728-96adb66e7b7d","Type":"ContainerStarted","Data":"e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765"}
Sep 11 18:07:49.779699 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.779643    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq" event={"ID":"ecf028d7-ebfa-4768-8728-96adb66e7b7d","Type":"ContainerStarted","Data":"7750f77cd5630849e0ef7ab541e4f8fe66ce40409b23273bcf9b02ef2a9e7d2d"}
Sep 11 18:07:49.779928 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:49.779866    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:07:50.445543 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:50.445506    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:50.445792 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:50.445771    2571 patch_prober.go:28] interesting pod/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 18:07:50.445875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:50.445825    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 18:07:50.797222 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:50.797116    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" event={"ID":"e966a2f4-4fcc-4478-93b8-13100af36dd8","Type":"ContainerStarted","Data":"b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f"}
Sep 11 18:07:50.828237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:50.828186    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq" podStartSLOduration=2.828171334 podStartE2EDuration="2.828171334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:49.855364479 +0000 UTC m=+833.625430828" watchObservedRunningTime="2026-09-11 18:07:50.828171334 +0000 UTC m=+834.598237694"
Sep 11 18:07:51.446092 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:51.446055    2571 patch_prober.go:28] interesting pod/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 18:07:51.446292 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:51.446114    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 18:07:51.525876 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:51.525842    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:51.526763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:51.526743    2571 patch_prober.go:28] interesting pod/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 18:07:51.526818 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:51.526790    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 18:07:52.445698 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:52.445657    2571 patch_prober.go:28] interesting pod/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 18:07:52.446113 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:52.445716    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 18:07:52.525614 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:52.525568    2571 patch_prober.go:28] interesting pod/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 18:07:52.525821 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:52.525639    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 18:07:53.445510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:53.445478    2571 patch_prober.go:28] interesting pod/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 18:07:53.445746 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:53.445725    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 18:07:53.525360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:53.525326    2571 patch_prober.go:28] interesting pod/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 18:07:53.525534 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:53.525380    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 18:07:54.446551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:54.446056    2571 patch_prober.go:28] interesting pod/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Sep 11 18:07:54.446551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:54.446102    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Sep 11 18:07:54.526113 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:54.526067    2571 patch_prober.go:28] interesting pod/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 18:07:54.526320 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:54.526144    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 18:07:55.463983 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:55.463944    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:55.465249 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:55.465221    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:55.471982 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:55.471963    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:07:55.499741 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:55.499708    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:55.525509 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:55.525477    2571 patch_prober.go:28] interesting pod/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Sep 11 18:07:55.525638 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:55.525532    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Sep 11 18:07:56.529738 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:56.529696    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:07:56.530849 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:07:56.530827    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:08:00.800637 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:00.800503    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:08:04.548457 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.547057    2571 patch_prober.go:28] interesting pod/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:45150->10.132.0.61:15021: read: connection reset by peer" start-of-body=
Sep 11 18:08:04.548457 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.547136    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:45150->10.132.0.61:15021: read: connection reset by peer"
Sep 11 18:08:04.622775 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.622746    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"]
Sep 11 18:08:04.625379 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.625363    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"
Sep 11 18:08:04.629350 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.629329    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:08:04.637050 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.637028    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"]
Sep 11 18:08:04.689126 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.689091    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60751aca-c6e3-42ba-8aab-090c56bd4c08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33958e97-k9hps\" (UID: \"60751aca-c6e3-42ba-8aab-090c56bd4c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"
Sep 11 18:08:04.790666 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.790626    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60751aca-c6e3-42ba-8aab-090c56bd4c08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33958e97-k9hps\" (UID: \"60751aca-c6e3-42ba-8aab-090c56bd4c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"
Sep 11 18:08:04.792977 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.791750    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60751aca-c6e3-42ba-8aab-090c56bd4c08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-33958e97-k9hps\" (UID: \"60751aca-c6e3-42ba-8aab-090c56bd4c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"
Sep 11 18:08:04.933981 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.933949    2571 generic.go:356] "Generic (PLEG): container finished" podID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerID="2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f" exitCode=0
Sep 11 18:08:04.934147 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.934023    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" event={"ID":"b41ba5e4-c105-4400-8cbb-90087a386a2a","Type":"ContainerDied","Data":"2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f"}
Sep 11 18:08:04.934578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.934557    2571 scope.go:117] "RemoveContainer" containerID="2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f"
Sep 11 18:08:04.935805 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.935773    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:08:04.935902 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.935844    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:08:04.935902 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.935873    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:08:04.936431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.936099    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"
Sep 11 18:08:04.936431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.936216    2571 generic.go:356] "Generic (PLEG): container finished" podID="c628866d-0242-49cd-b3ef-766ad525bfce" containerID="61585775a3e7c365c7361d4eba735ac4674ad459e28ba33e0f30ec959707cda1" exitCode=0
Sep 11 18:08:04.936431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.936275    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" event={"ID":"c628866d-0242-49cd-b3ef-766ad525bfce","Type":"ContainerDied","Data":"61585775a3e7c365c7361d4eba735ac4674ad459e28ba33e0f30ec959707cda1"}
Sep 11 18:08:04.936680 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.936629    2571 scope.go:117] "RemoveContainer" containerID="61585775a3e7c365c7361d4eba735ac4674ad459e28ba33e0f30ec959707cda1"
Sep 11 18:08:04.937758 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.937730    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:08:04.937849 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.937800    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:08:04.937907 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:04.937855    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:08:05.230799 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:08:05.230769    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60751aca_c6e3_42ba_8aab_090c56bd4c08.slice/crio-32c573225bd5e2952fb2a405eac40bb9692cbe8ba562a5e4e6db5dc0276bbb84 WatchSource:0}: Error finding container 32c573225bd5e2952fb2a405eac40bb9692cbe8ba562a5e4e6db5dc0276bbb84: Status 404 returned error can't find the container with id 32c573225bd5e2952fb2a405eac40bb9692cbe8ba562a5e4e6db5dc0276bbb84
Sep 11 18:08:05.232395 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:05.232370    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"]
Sep 11 18:08:05.943514 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:05.943478    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps" event={"ID":"60751aca-c6e3-42ba-8aab-090c56bd4c08","Type":"ContainerStarted","Data":"32c573225bd5e2952fb2a405eac40bb9692cbe8ba562a5e4e6db5dc0276bbb84"}
Sep 11 18:08:05.945383 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:05.945364    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" event={"ID":"b41ba5e4-c105-4400-8cbb-90087a386a2a","Type":"ContainerStarted","Data":"f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df"}
Sep 11 18:08:05.947186 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:05.947167    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" event={"ID":"c628866d-0242-49cd-b3ef-766ad525bfce","Type":"ContainerStarted","Data":"ea47e7f2f60ecee507fad6171ed0b97b23385f92b3090ecb901d1f1a2aeb7166"}
Sep 11 18:08:06.210323 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:06.210217    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:08:06.210464 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:06.210401    2571 patch_prober.go:28] interesting pod/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 18:08:06.210512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:06.210452    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 18:08:06.444140 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:06.444110    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:08:06.444365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:06.444335    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 18:08:06.444415 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:06.444394    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 18:08:07.210029 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:07.209990    2571 patch_prober.go:28] interesting pod/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 18:08:07.210493 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:07.210074    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 18:08:07.444666 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:07.444522    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 18:08:07.444666 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:07.444579    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 18:08:08.209608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:08.209577    2571 patch_prober.go:28] interesting pod/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 18:08:08.209795 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:08.209634    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 18:08:08.444193 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:08.444151    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 18:08:08.444599 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:08.444218    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 18:08:09.210075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:09.210038    2571 patch_prober.go:28] interesting pod/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 18:08:09.210239 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:09.210103    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 18:08:09.444670 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:09.444605    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 18:08:09.445134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:09.444696    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 18:08:10.209962 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:10.209924    2571 patch_prober.go:28] interesting pod/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 18:08:10.210076 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:10.209980    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 18:08:10.444449 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:10.444414    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 18:08:10.444620 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:10.444468    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 18:08:11.019703 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:11.019599    2571 generic.go:356] "Generic (PLEG): container finished" podID="60751aca-c6e3-42ba-8aab-090c56bd4c08" containerID="b060777644a1796c3f90bd734eda6145889dfb5b129797fd3f9fe6f3f3bfef28" exitCode=0
Sep 11 18:08:11.019703 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:11.019662    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps" event={"ID":"60751aca-c6e3-42ba-8aab-090c56bd4c08","Type":"ContainerDied","Data":"b060777644a1796c3f90bd734eda6145889dfb5b129797fd3f9fe6f3f3bfef28"}
Sep 11 18:08:11.209879 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:11.209836    2571 patch_prober.go:28] interesting pod/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 18:08:11.210041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:11.209927    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 18:08:11.444399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:11.444361    2571 patch_prober.go:28] interesting pod/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 11 18:08:11.444568 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:11.444431    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 11 18:08:12.210308 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.210101    2571 patch_prober.go:28] interesting pod/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Sep 11 18:08:12.210308 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.210158    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Sep 11 18:08:12.259013 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.258982    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"
Sep 11 18:08:12.375379 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.375131    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60751aca-c6e3-42ba-8aab-090c56bd4c08-maas-api-service-ca\") pod \"60751aca-c6e3-42ba-8aab-090c56bd4c08\" (UID: \"60751aca-c6e3-42ba-8aab-090c56bd4c08\") "
Sep 11 18:08:12.375595 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.375567    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60751aca-c6e3-42ba-8aab-090c56bd4c08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60751aca-c6e3-42ba-8aab-090c56bd4c08" (UID: "60751aca-c6e3-42ba-8aab-090c56bd4c08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:08:12.448498 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.448458    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:08:12.448901 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.448877    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:08:12.449582 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.449561    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:08:12.476618 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:12.476580    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60751aca-c6e3-42ba-8aab-090c56bd4c08-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:13.034321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:13.034216    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"
Sep 11 18:08:13.034321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:13.034231    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps" event={"ID":"60751aca-c6e3-42ba-8aab-090c56bd4c08","Type":"ContainerDied","Data":"32c573225bd5e2952fb2a405eac40bb9692cbe8ba562a5e4e6db5dc0276bbb84"}
Sep 11 18:08:13.034321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:13.034289    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c573225bd5e2952fb2a405eac40bb9692cbe8ba562a5e4e6db5dc0276bbb84"
Sep 11 18:08:13.213694 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:13.213659    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:08:13.214081 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:13.213925    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:08:13.214578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:13.214556    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:08:17.547739 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:17.547706    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"]
Sep 11 18:08:17.555426 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:17.555402    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-33958e97-k9hps"]
Sep 11 18:08:17.753099 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:17.753053    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"]
Sep 11 18:08:17.753308 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:17.753268    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy" containerID="cri-o://f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df" gracePeriod=30
Sep 11 18:08:18.731084 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:18.731055    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60751aca-c6e3-42ba-8aab-090c56bd4c08" path="/var/lib/kubelet/pods/60751aca-c6e3-42ba-8aab-090c56bd4c08/volumes"
Sep 11 18:08:22.990832 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:22.990803    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:08:23.081865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.081781    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-data\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.081865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.081836    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlqgs\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-kube-api-access-hlqgs\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.081865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.081856    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-podinfo\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.082114 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.081884    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-envoy\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.082114 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.081907    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-certs\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.082114 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.081925    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-token\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.082114 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.081969    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41ba5e4-c105-4400-8cbb-90087a386a2a-istiod-ca-cert\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.082356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082167    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-data" (OuterVolumeSpecName: "istio-data") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:23.082356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082281    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:23.082489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082458    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41ba5e4-c105-4400-8cbb-90087a386a2a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:08:23.082489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082480    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-socket\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.082603 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082532    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-credential-socket\") pod \"b41ba5e4-c105-4400-8cbb-90087a386a2a\" (UID: \"b41ba5e4-c105-4400-8cbb-90087a386a2a\") "
Sep 11 18:08:23.082603 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082482    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:23.082840 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082817    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:23.082966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082847    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:23.082966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082861    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b41ba5e4-c105-4400-8cbb-90087a386a2a-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:23.082966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082874    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:23.082966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.082816    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:23.084082 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.084061    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-kube-api-access-hlqgs" (OuterVolumeSpecName: "kube-api-access-hlqgs") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "kube-api-access-hlqgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:08:23.084154 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.084138    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:08:23.084470 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.084450    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:08:23.084583 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.084558    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b41ba5e4-c105-4400-8cbb-90087a386a2a" (UID: "b41ba5e4-c105-4400-8cbb-90087a386a2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:08:23.099364 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.099336    2571 generic.go:356] "Generic (PLEG): container finished" podID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerID="f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df" exitCode=0
Sep 11 18:08:23.099459 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.099389    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"
Sep 11 18:08:23.099459 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.099421    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" event={"ID":"b41ba5e4-c105-4400-8cbb-90087a386a2a","Type":"ContainerDied","Data":"f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df"}
Sep 11 18:08:23.099539 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.099480    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx" event={"ID":"b41ba5e4-c105-4400-8cbb-90087a386a2a","Type":"ContainerDied","Data":"781a85092549d882eab87245b4f61a55cef253f051d375993da3ce54a9c783bf"}
Sep 11 18:08:23.099539 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.099510    2571 scope.go:117] "RemoveContainer" containerID="f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df"
Sep 11 18:08:23.109136 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.109119    2571 scope.go:117] "RemoveContainer" containerID="2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f"
Sep 11 18:08:23.117613 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.117596    2571 scope.go:117] "RemoveContainer" containerID="f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df"
Sep 11 18:08:23.117867 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:08:23.117849    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df\": container with ID starting with f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df not found: ID does not exist" containerID="f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df"
Sep 11 18:08:23.117917 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.117875    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df"} err="failed to get container status \"f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df\": rpc error: code = NotFound desc = could not find container \"f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df\": container with ID starting with f91a758b2964fedefbda2a2f1438dbb1dcac790d472a88eb56c1f825055c56df not found: ID does not exist"
Sep 11 18:08:23.117917 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.117893    2571 scope.go:117] "RemoveContainer" containerID="2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f"
Sep 11 18:08:23.118154 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:08:23.118136    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f\": container with ID starting with 2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f not found: ID does not exist" containerID="2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f"
Sep 11 18:08:23.118210 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.118159    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f"} err="failed to get container status \"2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f\": rpc error: code = NotFound desc = could not find container \"2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f\": container with ID starting with 2c16e75b8ed7823004928906dcd30fb6939d380fb22d38dd659e7902a348585f not found: ID does not exist"
Sep 11 18:08:23.122716 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.122696    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"]
Sep 11 18:08:23.126729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.126710    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-33958e97-openshift-default-6cdb6877d-mbtmx"]
Sep 11 18:08:23.183613 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.183580    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:23.183613 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.183610    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlqgs\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-kube-api-access-hlqgs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:23.183613 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.183621    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:23.183846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.183630    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:23.183846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:23.183638    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b41ba5e4-c105-4400-8cbb-90087a386a2a-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:24.731532 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:24.731493    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" path="/var/lib/kubelet/pods/b41ba5e4-c105-4400-8cbb-90087a386a2a/volumes"
Sep 11 18:08:25.505308 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.505273    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"]
Sep 11 18:08:25.505702 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.505690    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy"
Sep 11 18:08:25.505747 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.505704    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy"
Sep 11 18:08:25.505747 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.505713    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60751aca-c6e3-42ba-8aab-090c56bd4c08" containerName="revoke-keys"
Sep 11 18:08:25.505747 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.505718    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="60751aca-c6e3-42ba-8aab-090c56bd4c08" containerName="revoke-keys"
Sep 11 18:08:25.508477 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.506232    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy"
Sep 11 18:08:25.508477 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.506339    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="60751aca-c6e3-42ba-8aab-090c56bd4c08" containerName="revoke-keys"
Sep 11 18:08:25.511838 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.511808    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:25.516698 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.516671    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-44ba3006\""
Sep 11 18:08:25.516972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.516950    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-44ba3006-dockercfg-rtwwl\""
Sep 11 18:08:25.518629 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.518605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"]
Sep 11 18:08:25.604324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.604129    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlvq\" (UniqueName: \"kubernetes.io/projected/0767796d-8759-4384-9b63-f0990583a09f-kube-api-access-mtlvq\") pod \"payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph\" (UID: \"0767796d-8759-4384-9b63-f0990583a09f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:25.604324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.604191    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0767796d-8759-4384-9b63-f0990583a09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph\" (UID: \"0767796d-8759-4384-9b63-f0990583a09f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:25.705260 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.705218    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlvq\" (UniqueName: \"kubernetes.io/projected/0767796d-8759-4384-9b63-f0990583a09f-kube-api-access-mtlvq\") pod \"payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph\" (UID: \"0767796d-8759-4384-9b63-f0990583a09f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:25.705260 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.705264    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0767796d-8759-4384-9b63-f0990583a09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph\" (UID: \"0767796d-8759-4384-9b63-f0990583a09f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:25.706016 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.705993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0767796d-8759-4384-9b63-f0990583a09f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph\" (UID: \"0767796d-8759-4384-9b63-f0990583a09f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:25.717945 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.717909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlvq\" (UniqueName: \"kubernetes.io/projected/0767796d-8759-4384-9b63-f0990583a09f-kube-api-access-mtlvq\") pod \"payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph\" (UID: \"0767796d-8759-4384-9b63-f0990583a09f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:25.826083 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:25.825995    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:26.099792 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:26.099765    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"]
Sep 11 18:08:26.102105 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:08:26.102080    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0767796d_8759_4384_9b63_f0990583a09f.slice/crio-f9f5c1895e8fcf04bc7e840d5c838b33ac9c81576d9da23bdadc0bfebb958a52 WatchSource:0}: Error finding container f9f5c1895e8fcf04bc7e840d5c838b33ac9c81576d9da23bdadc0bfebb958a52: Status 404 returned error can't find the container with id f9f5c1895e8fcf04bc7e840d5c838b33ac9c81576d9da23bdadc0bfebb958a52
Sep 11 18:08:26.119588 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:26.119564    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph" event={"ID":"0767796d-8759-4384-9b63-f0990583a09f","Type":"ContainerStarted","Data":"f9f5c1895e8fcf04bc7e840d5c838b33ac9c81576d9da23bdadc0bfebb958a52"}
Sep 11 18:08:27.126918 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:27.126881    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph" event={"ID":"0767796d-8759-4384-9b63-f0990583a09f","Type":"ContainerStarted","Data":"6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb"}
Sep 11 18:08:27.127321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:27.126960    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:27.151897 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:27.151852    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph" podStartSLOduration=2.151835943 podStartE2EDuration="2.151835943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:27.151200276 +0000 UTC m=+870.921266625" watchObservedRunningTime="2026-09-11 18:08:27.151835943 +0000 UTC m=+870.921902284"
Sep 11 18:08:38.134948 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:38.134915    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:08:52.042344 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.042306    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"]
Sep 11 18:08:52.042753 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.042742    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy"
Sep 11 18:08:52.042753 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.042753    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy"
Sep 11 18:08:52.042830 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.042818    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b41ba5e4-c105-4400-8cbb-90087a386a2a" containerName="istio-proxy"
Sep 11 18:08:52.049422 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.049396    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"
Sep 11 18:08:52.053104 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.053077    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:08:52.072006 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.071979    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"]
Sep 11 18:08:52.140058 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.140027    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912d4ae5-74c2-4cd2-a43a-e48f6079c609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6\" (UID: \"912d4ae5-74c2-4cd2-a43a-e48f6079c609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"
Sep 11 18:08:52.241319 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.241278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912d4ae5-74c2-4cd2-a43a-e48f6079c609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6\" (UID: \"912d4ae5-74c2-4cd2-a43a-e48f6079c609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"
Sep 11 18:08:52.241943 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.241922    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912d4ae5-74c2-4cd2-a43a-e48f6079c609-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6\" (UID: \"912d4ae5-74c2-4cd2-a43a-e48f6079c609\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"
Sep 11 18:08:52.359389 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.359289    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"
Sep 11 18:08:52.631241 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:52.631210    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"]
Sep 11 18:08:52.632538 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:08:52.632515    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912d4ae5_74c2_4cd2_a43a_e48f6079c609.slice/crio-5e910eb54400a47c84dfe147d8a598c1f8e783128c0dd71eb52324e4f9dee736 WatchSource:0}: Error finding container 5e910eb54400a47c84dfe147d8a598c1f8e783128c0dd71eb52324e4f9dee736: Status 404 returned error can't find the container with id 5e910eb54400a47c84dfe147d8a598c1f8e783128c0dd71eb52324e4f9dee736
Sep 11 18:08:53.294541 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:53.294504    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6" event={"ID":"912d4ae5-74c2-4cd2-a43a-e48f6079c609","Type":"ContainerStarted","Data":"e47b77997586eb4a2a4c3585dc2b23283d7eb07a26b6c58795c65abf20f4a0b7"}
Sep 11 18:08:53.294925 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:53.294557    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6" event={"ID":"912d4ae5-74c2-4cd2-a43a-e48f6079c609","Type":"ContainerStarted","Data":"5e910eb54400a47c84dfe147d8a598c1f8e783128c0dd71eb52324e4f9dee736"}
Sep 11 18:08:53.314977 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:53.314926    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6" podStartSLOduration=1.31491015 podStartE2EDuration="1.31491015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:53.31201093 +0000 UTC m=+897.082077279" watchObservedRunningTime="2026-09-11 18:08:53.31491015 +0000 UTC m=+897.084976498"
Sep 11 18:08:54.301531 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:54.301495    2571 generic.go:356] "Generic (PLEG): container finished" podID="912d4ae5-74c2-4cd2-a43a-e48f6079c609" containerID="e47b77997586eb4a2a4c3585dc2b23283d7eb07a26b6c58795c65abf20f4a0b7" exitCode=0
Sep 11 18:08:54.301942 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:54.301583    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6" event={"ID":"912d4ae5-74c2-4cd2-a43a-e48f6079c609","Type":"ContainerDied","Data":"e47b77997586eb4a2a4c3585dc2b23283d7eb07a26b6c58795c65abf20f4a0b7"}
Sep 11 18:08:55.469111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:55.469085    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"
Sep 11 18:08:55.577282 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:55.577194    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912d4ae5-74c2-4cd2-a43a-e48f6079c609-maas-api-service-ca\") pod \"912d4ae5-74c2-4cd2-a43a-e48f6079c609\" (UID: \"912d4ae5-74c2-4cd2-a43a-e48f6079c609\") "
Sep 11 18:08:55.577564 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:55.577542    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912d4ae5-74c2-4cd2-a43a-e48f6079c609-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "912d4ae5-74c2-4cd2-a43a-e48f6079c609" (UID: "912d4ae5-74c2-4cd2-a43a-e48f6079c609"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:08:55.678252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:55.678216    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/912d4ae5-74c2-4cd2-a43a-e48f6079c609-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:08:56.314945 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:56.314915    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"
Sep 11 18:08:56.315120 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:56.314916    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6" event={"ID":"912d4ae5-74c2-4cd2-a43a-e48f6079c609","Type":"ContainerDied","Data":"5e910eb54400a47c84dfe147d8a598c1f8e783128c0dd71eb52324e4f9dee736"}
Sep 11 18:08:56.315120 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:08:56.315035    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e910eb54400a47c84dfe147d8a598c1f8e783128c0dd71eb52324e4f9dee736"
Sep 11 18:09:04.424509 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.424460    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"]
Sep 11 18:09:04.425791 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.425737    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph" podUID="0767796d-8759-4384-9b63-f0990583a09f" containerName="payload-pre-processing" containerID="cri-o://6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb" gracePeriod=30
Sep 11 18:09:04.700816 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.700793    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:09:04.738537 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.738504    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"]
Sep 11 18:09:04.740966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.740943    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-44ba3006-mwkh6"]
Sep 11 18:09:04.767985 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.767958    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtlvq\" (UniqueName: \"kubernetes.io/projected/0767796d-8759-4384-9b63-f0990583a09f-kube-api-access-mtlvq\") pod \"0767796d-8759-4384-9b63-f0990583a09f\" (UID: \"0767796d-8759-4384-9b63-f0990583a09f\") "
Sep 11 18:09:04.768111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.768022    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0767796d-8759-4384-9b63-f0990583a09f-plugins-config-volume\") pod \"0767796d-8759-4384-9b63-f0990583a09f\" (UID: \"0767796d-8759-4384-9b63-f0990583a09f\") "
Sep 11 18:09:04.768416 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.768391    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0767796d-8759-4384-9b63-f0990583a09f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0767796d-8759-4384-9b63-f0990583a09f" (UID: "0767796d-8759-4384-9b63-f0990583a09f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:09:04.770045 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.770023    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0767796d-8759-4384-9b63-f0990583a09f-kube-api-access-mtlvq" (OuterVolumeSpecName: "kube-api-access-mtlvq") pod "0767796d-8759-4384-9b63-f0990583a09f" (UID: "0767796d-8759-4384-9b63-f0990583a09f"). InnerVolumeSpecName "kube-api-access-mtlvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:04.868880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.868845    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlvq\" (UniqueName: \"kubernetes.io/projected/0767796d-8759-4384-9b63-f0990583a09f-kube-api-access-mtlvq\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:04.868880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:04.868874    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0767796d-8759-4384-9b63-f0990583a09f-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:05.373982 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.373940    2571 generic.go:356] "Generic (PLEG): container finished" podID="0767796d-8759-4384-9b63-f0990583a09f" containerID="6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb" exitCode=0
Sep 11 18:09:05.374189 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.373994    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"
Sep 11 18:09:05.374189 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.374027    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph" event={"ID":"0767796d-8759-4384-9b63-f0990583a09f","Type":"ContainerDied","Data":"6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb"}
Sep 11 18:09:05.374189 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.374078    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph" event={"ID":"0767796d-8759-4384-9b63-f0990583a09f","Type":"ContainerDied","Data":"f9f5c1895e8fcf04bc7e840d5c838b33ac9c81576d9da23bdadc0bfebb958a52"}
Sep 11 18:09:05.374189 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.374101    2571 scope.go:117] "RemoveContainer" containerID="6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb"
Sep 11 18:09:05.383595 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.383578    2571 scope.go:117] "RemoveContainer" containerID="6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb"
Sep 11 18:09:05.383868 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:09:05.383851    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb\": container with ID starting with 6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb not found: ID does not exist" containerID="6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb"
Sep 11 18:09:05.383925 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.383877    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb"} err="failed to get container status \"6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb\": rpc error: code = NotFound desc = could not find container \"6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb\": container with ID starting with 6dee04b77488165637ff5761428184f97c1bfefff21fc56e987183b43bb68fdb not found: ID does not exist"
Sep 11 18:09:05.403356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.403323    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"]
Sep 11 18:09:05.405834 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.405811    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-44ba3006-5cd897666-4nzph"]
Sep 11 18:09:05.799243 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.799214    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"]
Sep 11 18:09:05.799709 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.799694    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0767796d-8759-4384-9b63-f0990583a09f" containerName="payload-pre-processing"
Sep 11 18:09:05.799753 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.799713    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0767796d-8759-4384-9b63-f0990583a09f" containerName="payload-pre-processing"
Sep 11 18:09:05.799753 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.799733    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="912d4ae5-74c2-4cd2-a43a-e48f6079c609" containerName="revoke-keys"
Sep 11 18:09:05.799753 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.799739    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="912d4ae5-74c2-4cd2-a43a-e48f6079c609" containerName="revoke-keys"
Sep 11 18:09:05.799853 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.799820    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0767796d-8759-4384-9b63-f0990583a09f" containerName="payload-pre-processing"
Sep 11 18:09:05.799853 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.799827    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="912d4ae5-74c2-4cd2-a43a-e48f6079c609" containerName="revoke-keys"
Sep 11 18:09:05.804273 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.804257    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.809615 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.809593    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-820e15b2-openshift-default-dockercfg-7n5hd\""
Sep 11 18:09:05.853729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.853702    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"]
Sep 11 18:09:05.878431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-data\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.878608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878443    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-socket\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.878608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878481    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-envoy\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.878608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878523    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-podinfo\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.878608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e735cbf-bcba-4c75-947a-9e15461b78b8-istiod-ca-cert\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.878832 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsktj\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-kube-api-access-xsktj\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.878832 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878688    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-credential-socket\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.878832 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878720    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-token\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.878832 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.878750    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-certs\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980158    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-data\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980211    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-socket\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980242    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-envoy\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-podinfo\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980293    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e735cbf-bcba-4c75-947a-9e15461b78b8-istiod-ca-cert\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980439    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsktj\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-kube-api-access-xsktj\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-credential-socket\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980541    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-token\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980572    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-certs\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980831 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980652    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-data\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.980831 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.980746    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-socket\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.981026 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.981004    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e735cbf-bcba-4c75-947a-9e15461b78b8-istiod-ca-cert\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.981083 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.981008    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-credential-socket\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.981083 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.981077    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-certs\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.982629 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.982609    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-envoy\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.982872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.982856    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-podinfo\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.996922 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.996895    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-token\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:05.997104 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:05.997086    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsktj\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-kube-api-access-xsktj\") pod \"e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") " pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:06.114266 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:06.114187    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:06.413651 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:06.413627    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"]
Sep 11 18:09:06.415510 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:09:06.415475    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e735cbf_bcba_4c75_947a_9e15461b78b8.slice/crio-fc1dfe8a4f4eeb51860806411d9ba37f9def6b27e307f72ff6601d694f3c0742 WatchSource:0}: Error finding container fc1dfe8a4f4eeb51860806411d9ba37f9def6b27e307f72ff6601d694f3c0742: Status 404 returned error can't find the container with id fc1dfe8a4f4eeb51860806411d9ba37f9def6b27e307f72ff6601d694f3c0742
Sep 11 18:09:06.417712 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:06.417682    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:09:06.417784 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:06.417744    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:09:06.417784 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:06.417772    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:09:06.732636 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:06.732555    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0767796d-8759-4384-9b63-f0990583a09f" path="/var/lib/kubelet/pods/0767796d-8759-4384-9b63-f0990583a09f/volumes"
Sep 11 18:09:06.733036 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:06.733017    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="912d4ae5-74c2-4cd2-a43a-e48f6079c609" path="/var/lib/kubelet/pods/912d4ae5-74c2-4cd2-a43a-e48f6079c609/volumes"
Sep 11 18:09:07.392956 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:07.392918    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" event={"ID":"0e735cbf-bcba-4c75-947a-9e15461b78b8","Type":"ContainerStarted","Data":"26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb"}
Sep 11 18:09:07.393385 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:07.392974    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" event={"ID":"0e735cbf-bcba-4c75-947a-9e15461b78b8","Type":"ContainerStarted","Data":"fc1dfe8a4f4eeb51860806411d9ba37f9def6b27e307f72ff6601d694f3c0742"}
Sep 11 18:09:07.437208 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:07.437158    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" podStartSLOduration=2.437142959 podStartE2EDuration="2.437142959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:07.436125488 +0000 UTC m=+911.206191836" watchObservedRunningTime="2026-09-11 18:09:07.437142959 +0000 UTC m=+911.207209308"
Sep 11 18:09:08.115323 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:08.115258    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:08.120548 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:08.120521    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:08.399520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:08.399436    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:08.400552 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:08.400530    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:29.582538 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:29.582505    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"]
Sep 11 18:09:29.586023 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:29.586005    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"
Sep 11 18:09:29.589129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:29.589107    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:09:29.599548 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:29.599517    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"]
Sep 11 18:09:29.705861 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:29.705808    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be157-1238-42f5-81f1-cc1c31561b7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-736ba3-wvjp8\" (UID: \"f39be157-1238-42f5-81f1-cc1c31561b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"
Sep 11 18:09:29.806561 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:29.806525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be157-1238-42f5-81f1-cc1c31561b7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-736ba3-wvjp8\" (UID: \"f39be157-1238-42f5-81f1-cc1c31561b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"
Sep 11 18:09:29.807139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:29.807117    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be157-1238-42f5-81f1-cc1c31561b7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-736ba3-wvjp8\" (UID: \"f39be157-1238-42f5-81f1-cc1c31561b7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"
Sep 11 18:09:29.899455 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:29.899356    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"
Sep 11 18:09:30.193633 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:30.193599    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"]
Sep 11 18:09:30.194073 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:09:30.194053    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39be157_1238_42f5_81f1_cc1c31561b7b.slice/crio-4f604e4e324a5267d3da5dfb53086997b31f52d24127439d51838853175b984d WatchSource:0}: Error finding container 4f604e4e324a5267d3da5dfb53086997b31f52d24127439d51838853175b984d: Status 404 returned error can't find the container with id 4f604e4e324a5267d3da5dfb53086997b31f52d24127439d51838853175b984d
Sep 11 18:09:30.542674 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:30.542639    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8" event={"ID":"f39be157-1238-42f5-81f1-cc1c31561b7b","Type":"ContainerStarted","Data":"54cc1113d998d327bcf4341dea812a4330fe381bb977bf2b98db93203fc8c9de"}
Sep 11 18:09:30.542810 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:30.542702    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8" event={"ID":"f39be157-1238-42f5-81f1-cc1c31561b7b","Type":"ContainerStarted","Data":"4f604e4e324a5267d3da5dfb53086997b31f52d24127439d51838853175b984d"}
Sep 11 18:09:30.565994 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:30.565924    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8" podStartSLOduration=1.565902585 podStartE2EDuration="1.565902585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:30.562480054 +0000 UTC m=+934.332546405" watchObservedRunningTime="2026-09-11 18:09:30.565902585 +0000 UTC m=+934.335968935"
Sep 11 18:09:31.557738 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:31.557634    2571 generic.go:356] "Generic (PLEG): container finished" podID="f39be157-1238-42f5-81f1-cc1c31561b7b" containerID="54cc1113d998d327bcf4341dea812a4330fe381bb977bf2b98db93203fc8c9de" exitCode=0
Sep 11 18:09:31.557738 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:31.557705    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8" event={"ID":"f39be157-1238-42f5-81f1-cc1c31561b7b","Type":"ContainerDied","Data":"54cc1113d998d327bcf4341dea812a4330fe381bb977bf2b98db93203fc8c9de"}
Sep 11 18:09:32.713865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:32.713840    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"
Sep 11 18:09:32.732176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:32.731880    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be157-1238-42f5-81f1-cc1c31561b7b-maas-api-service-ca\") pod \"f39be157-1238-42f5-81f1-cc1c31561b7b\" (UID: \"f39be157-1238-42f5-81f1-cc1c31561b7b\") "
Sep 11 18:09:32.732373 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:32.732242    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39be157-1238-42f5-81f1-cc1c31561b7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f39be157-1238-42f5-81f1-cc1c31561b7b" (UID: "f39be157-1238-42f5-81f1-cc1c31561b7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:09:32.833388 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:32.833311    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f39be157-1238-42f5-81f1-cc1c31561b7b-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:33.572775 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:33.572743    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"
Sep 11 18:09:33.572981 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:33.572779    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8" event={"ID":"f39be157-1238-42f5-81f1-cc1c31561b7b","Type":"ContainerDied","Data":"4f604e4e324a5267d3da5dfb53086997b31f52d24127439d51838853175b984d"}
Sep 11 18:09:33.572981 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:33.572826    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f604e4e324a5267d3da5dfb53086997b31f52d24127439d51838853175b984d"
Sep 11 18:09:34.567105 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.567070    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"]
Sep 11 18:09:34.569438 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.567749    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39be157-1238-42f5-81f1-cc1c31561b7b" containerName="revoke-keys"
Sep 11 18:09:34.569438 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.567765    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39be157-1238-42f5-81f1-cc1c31561b7b" containerName="revoke-keys"
Sep 11 18:09:34.569438 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.567881    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39be157-1238-42f5-81f1-cc1c31561b7b" containerName="revoke-keys"
Sep 11 18:09:34.571327 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.571310    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"
Sep 11 18:09:34.574709 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.574687    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:09:34.580083 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.580023    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"]
Sep 11 18:09:34.650612 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.650576    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e72837-3216-4d56-91f0-3f26288d769e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-820e15b2-7w466\" (UID: \"17e72837-3216-4d56-91f0-3f26288d769e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"
Sep 11 18:09:34.751852 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.751813    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e72837-3216-4d56-91f0-3f26288d769e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-820e15b2-7w466\" (UID: \"17e72837-3216-4d56-91f0-3f26288d769e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"
Sep 11 18:09:34.752556 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.752534    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e72837-3216-4d56-91f0-3f26288d769e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-820e15b2-7w466\" (UID: \"17e72837-3216-4d56-91f0-3f26288d769e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"
Sep 11 18:09:34.886253 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:34.886142    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"
Sep 11 18:09:35.158623 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.158589    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"]
Sep 11 18:09:35.160056 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:09:35.160029    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e72837_3216_4d56_91f0_3f26288d769e.slice/crio-90aefaa429222c10a23f30541c4ec9c41b88e39e99847602156ccc5d836f9a0c WatchSource:0}: Error finding container 90aefaa429222c10a23f30541c4ec9c41b88e39e99847602156ccc5d836f9a0c: Status 404 returned error can't find the container with id 90aefaa429222c10a23f30541c4ec9c41b88e39e99847602156ccc5d836f9a0c
Sep 11 18:09:35.589087 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.589034    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466" event={"ID":"17e72837-3216-4d56-91f0-3f26288d769e","Type":"ContainerStarted","Data":"eeba4717353c4ad5d9eb11b3a2285b5e1be28661364b7ca47fe17bfbe7ea6273"}
Sep 11 18:09:35.589525 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.589117    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466" event={"ID":"17e72837-3216-4d56-91f0-3f26288d769e","Type":"ContainerStarted","Data":"90aefaa429222c10a23f30541c4ec9c41b88e39e99847602156ccc5d836f9a0c"}
Sep 11 18:09:35.590553 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.590529    2571 generic.go:356] "Generic (PLEG): container finished" podID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerID="26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb" exitCode=0
Sep 11 18:09:35.590671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.590577    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" event={"ID":"0e735cbf-bcba-4c75-947a-9e15461b78b8","Type":"ContainerDied","Data":"26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb"}
Sep 11 18:09:35.590957 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.590940    2571 scope.go:117] "RemoveContainer" containerID="26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb"
Sep 11 18:09:35.592067 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.592040    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:09:35.592138 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.592105    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:09:35.592178 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.592141    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:09:35.606525 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:35.606486    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466" podStartSLOduration=1.606472511 podStartE2EDuration="1.606472511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:35.605335963 +0000 UTC m=+939.375402310" watchObservedRunningTime="2026-09-11 18:09:35.606472511 +0000 UTC m=+939.376538862"
Sep 11 18:09:36.598093 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:36.598062    2571 generic.go:356] "Generic (PLEG): container finished" podID="17e72837-3216-4d56-91f0-3f26288d769e" containerID="eeba4717353c4ad5d9eb11b3a2285b5e1be28661364b7ca47fe17bfbe7ea6273" exitCode=0
Sep 11 18:09:36.598634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:36.598142    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466" event={"ID":"17e72837-3216-4d56-91f0-3f26288d769e","Type":"ContainerDied","Data":"eeba4717353c4ad5d9eb11b3a2285b5e1be28661364b7ca47fe17bfbe7ea6273"}
Sep 11 18:09:36.600176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:36.600155    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" event={"ID":"0e735cbf-bcba-4c75-947a-9e15461b78b8","Type":"ContainerStarted","Data":"7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e"}
Sep 11 18:09:37.115205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:37.115164    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:37.116224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:37.116198    2571 patch_prober.go:28] interesting pod/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 18:09:37.116359 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:37.116247    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 18:09:37.801802 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:37.801780    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"
Sep 11 18:09:37.885277 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:37.885196    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e72837-3216-4d56-91f0-3f26288d769e-maas-api-service-ca\") pod \"17e72837-3216-4d56-91f0-3f26288d769e\" (UID: \"17e72837-3216-4d56-91f0-3f26288d769e\") "
Sep 11 18:09:37.885667 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:37.885645    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e72837-3216-4d56-91f0-3f26288d769e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17e72837-3216-4d56-91f0-3f26288d769e" (UID: "17e72837-3216-4d56-91f0-3f26288d769e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:09:37.986242 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:37.986208    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17e72837-3216-4d56-91f0-3f26288d769e-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:38.115566 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:38.115521    2571 patch_prober.go:28] interesting pod/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 18:09:38.115769 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:38.115594    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 18:09:38.399909 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:38.399877    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:38.613478 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:38.613449    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"
Sep 11 18:09:38.613649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:38.613493    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466" event={"ID":"17e72837-3216-4d56-91f0-3f26288d769e","Type":"ContainerDied","Data":"90aefaa429222c10a23f30541c4ec9c41b88e39e99847602156ccc5d836f9a0c"}
Sep 11 18:09:38.613649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:38.613543    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90aefaa429222c10a23f30541c4ec9c41b88e39e99847602156ccc5d836f9a0c"
Sep 11 18:09:39.114809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:39.114771    2571 patch_prober.go:28] interesting pod/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 18:09:39.115279 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:39.114830    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 18:09:40.115517 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:40.115479    2571 patch_prober.go:28] interesting pod/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 18:09:40.115920 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:40.115534    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 18:09:41.115112 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.115079    2571 patch_prober.go:28] interesting pod/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 11 18:09:41.115319 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.115131    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 11 18:09:41.495221 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.495190    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"]
Sep 11 18:09:41.495696 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.495527    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l" podUID="62556da9-496e-42aa-85f2-1744fc2791e2" containerName="payload-processing" containerID="cri-o://2fb65d4281108edfbe9c9b40677726faa1ee81d9345ba8f7df31f080e52fedc8" gracePeriod=30
Sep 11 18:09:41.636060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.635887    2571 generic.go:356] "Generic (PLEG): container finished" podID="62556da9-496e-42aa-85f2-1744fc2791e2" containerID="2fb65d4281108edfbe9c9b40677726faa1ee81d9345ba8f7df31f080e52fedc8" exitCode=0
Sep 11 18:09:41.636060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.635948    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l" event={"ID":"62556da9-496e-42aa-85f2-1744fc2791e2","Type":"ContainerDied","Data":"2fb65d4281108edfbe9c9b40677726faa1ee81d9345ba8f7df31f080e52fedc8"}
Sep 11 18:09:41.804063 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.804021    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"]
Sep 11 18:09:41.806845 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.806820    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-736ba3-wvjp8"]
Sep 11 18:09:41.806971 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.806827    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:09:41.924373 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.924334    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf7rb\" (UniqueName: \"kubernetes.io/projected/62556da9-496e-42aa-85f2-1744fc2791e2-kube-api-access-rf7rb\") pod \"62556da9-496e-42aa-85f2-1744fc2791e2\" (UID: \"62556da9-496e-42aa-85f2-1744fc2791e2\") "
Sep 11 18:09:41.924564 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.924411    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62556da9-496e-42aa-85f2-1744fc2791e2-plugins-config-volume\") pod \"62556da9-496e-42aa-85f2-1744fc2791e2\" (UID: \"62556da9-496e-42aa-85f2-1744fc2791e2\") "
Sep 11 18:09:41.924793 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.924760    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62556da9-496e-42aa-85f2-1744fc2791e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62556da9-496e-42aa-85f2-1744fc2791e2" (UID: "62556da9-496e-42aa-85f2-1744fc2791e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:09:41.926496 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:41.926469    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62556da9-496e-42aa-85f2-1744fc2791e2-kube-api-access-rf7rb" (OuterVolumeSpecName: "kube-api-access-rf7rb") pod "62556da9-496e-42aa-85f2-1744fc2791e2" (UID: "62556da9-496e-42aa-85f2-1744fc2791e2"). InnerVolumeSpecName "kube-api-access-rf7rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:42.026135 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.026105    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf7rb\" (UniqueName: \"kubernetes.io/projected/62556da9-496e-42aa-85f2-1744fc2791e2-kube-api-access-rf7rb\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:42.026135 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.026131    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62556da9-496e-42aa-85f2-1744fc2791e2-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:42.254851 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.254808    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"]
Sep 11 18:09:42.262123 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.262097    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-820e15b2-7w466"]
Sep 11 18:09:42.296993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.296955    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:42.298130 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.298104    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:42.643342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.643237    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"
Sep 11 18:09:42.643342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.643280    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l" event={"ID":"62556da9-496e-42aa-85f2-1744fc2791e2","Type":"ContainerDied","Data":"7339591fb264229b5887cd5c48d5d99c8dcb9ef0e3a94a4fae87fcfbe19d7b8f"}
Sep 11 18:09:42.643840 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.643366    2571 scope.go:117] "RemoveContainer" containerID="2fb65d4281108edfbe9c9b40677726faa1ee81d9345ba8f7df31f080e52fedc8"
Sep 11 18:09:42.679903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.679878    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"]
Sep 11 18:09:42.684610 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.684589    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-736ba3-c754bcfbf-9dp4l"]
Sep 11 18:09:42.730763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.730732    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e72837-3216-4d56-91f0-3f26288d769e" path="/var/lib/kubelet/pods/17e72837-3216-4d56-91f0-3f26288d769e/volumes"
Sep 11 18:09:42.731110 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.731097    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62556da9-496e-42aa-85f2-1744fc2791e2" path="/var/lib/kubelet/pods/62556da9-496e-42aa-85f2-1744fc2791e2/volumes"
Sep 11 18:09:42.731416 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:42.731403    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39be157-1238-42f5-81f1-cc1c31561b7b" path="/var/lib/kubelet/pods/f39be157-1238-42f5-81f1-cc1c31561b7b/volumes"
Sep 11 18:09:43.352158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:43.352119    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"]
Sep 11 18:09:43.650862 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:43.650760    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy" containerID="cri-o://7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e" gracePeriod=30
Sep 11 18:09:48.836489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.825020    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"]
Sep 11 18:09:48.836489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.825663    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62556da9-496e-42aa-85f2-1744fc2791e2" containerName="payload-processing"
Sep 11 18:09:48.836489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.825680    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="62556da9-496e-42aa-85f2-1744fc2791e2" containerName="payload-processing"
Sep 11 18:09:48.836489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.825692    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e72837-3216-4d56-91f0-3f26288d769e" containerName="revoke-keys"
Sep 11 18:09:48.836489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.825701    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e72837-3216-4d56-91f0-3f26288d769e" containerName="revoke-keys"
Sep 11 18:09:48.836489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.825819    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="62556da9-496e-42aa-85f2-1744fc2791e2" containerName="payload-processing"
Sep 11 18:09:48.836489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.825833    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e72837-3216-4d56-91f0-3f26288d769e" containerName="revoke-keys"
Sep 11 18:09:48.846539 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.836596    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:48.851811 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.851787    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-85c5a226-dockercfg-ct78r\""
Sep 11 18:09:48.852221 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.852203    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-85c5a226\""
Sep 11 18:09:48.856651 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.856127    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"]
Sep 11 18:09:48.889871 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.889499    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzdx\" (UniqueName: \"kubernetes.io/projected/9b2c16f0-27c7-4b54-a078-29552a7c2c86-kube-api-access-4wzdx\") pod \"payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76\" (UID: \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:48.889871 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.889570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2c16f0-27c7-4b54-a078-29552a7c2c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76\" (UID: \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:48.942214 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.942188    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:48.990866 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.990838    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-certs\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991071 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.990878    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-podinfo\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991071 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.990911    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-token\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991112    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e735cbf-bcba-4c75-947a-9e15461b78b8-istiod-ca-cert\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991248 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991202    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-data\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991261    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-socket\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991114    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:48.991458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991334    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsktj\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-kube-api-access-xsktj\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991355    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e735cbf-bcba-4c75-947a-9e15461b78b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:09:48.991458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991362    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-credential-socket\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991406    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-envoy\") pod \"0e735cbf-bcba-4c75-947a-9e15461b78b8\" (UID: \"0e735cbf-bcba-4c75-947a-9e15461b78b8\") "
Sep 11 18:09:48.991649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991499    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:48.991649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991572    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:48.991761 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991678    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:48.991761 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzdx\" (UniqueName: \"kubernetes.io/projected/9b2c16f0-27c7-4b54-a078-29552a7c2c86-kube-api-access-4wzdx\") pod \"payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76\" (UID: \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:48.991869 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991791    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2c16f0-27c7-4b54-a078-29552a7c2c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76\" (UID: \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:48.991971 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991950    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:48.992035 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991976    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e735cbf-bcba-4c75-947a-9e15461b78b8-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:48.992035 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.991989    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:48.992035 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.992020    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:48.992226 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.992035    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:48.992800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.992772    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2c16f0-27c7-4b54-a078-29552a7c2c86-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76\" (UID: \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:48.993559 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.993503    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:48.993761 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.993729    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:09:48.993940 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.993920    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:09:48.994234 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:48.994211    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-kube-api-access-xsktj" (OuterVolumeSpecName: "kube-api-access-xsktj") pod "0e735cbf-bcba-4c75-947a-9e15461b78b8" (UID: "0e735cbf-bcba-4c75-947a-9e15461b78b8"). InnerVolumeSpecName "kube-api-access-xsktj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:49.002593 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.002571    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzdx\" (UniqueName: \"kubernetes.io/projected/9b2c16f0-27c7-4b54-a078-29552a7c2c86-kube-api-access-4wzdx\") pod \"payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76\" (UID: \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:49.092748 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.092714    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:49.092748 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.092742    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:49.092748 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.092754    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsktj\" (UniqueName: \"kubernetes.io/projected/0e735cbf-bcba-4c75-947a-9e15461b78b8-kube-api-access-xsktj\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:49.093011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.092767    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e735cbf-bcba-4c75-947a-9e15461b78b8-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:09:49.168796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.168761    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:49.449855 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.449814    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"]
Sep 11 18:09:49.450584 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:09:49.450556    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2c16f0_27c7_4b54_a078_29552a7c2c86.slice/crio-ffa7ace6c0fb01192ffa9805b28962da0dbeee8f999b1359b1386eaa41312050 WatchSource:0}: Error finding container ffa7ace6c0fb01192ffa9805b28962da0dbeee8f999b1359b1386eaa41312050: Status 404 returned error can't find the container with id ffa7ace6c0fb01192ffa9805b28962da0dbeee8f999b1359b1386eaa41312050
Sep 11 18:09:49.691063 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.691030    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76" event={"ID":"9b2c16f0-27c7-4b54-a078-29552a7c2c86","Type":"ContainerStarted","Data":"070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa"}
Sep 11 18:09:49.691241 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.691086    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76" event={"ID":"9b2c16f0-27c7-4b54-a078-29552a7c2c86","Type":"ContainerStarted","Data":"ffa7ace6c0fb01192ffa9805b28962da0dbeee8f999b1359b1386eaa41312050"}
Sep 11 18:09:49.691241 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.691225    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:09:49.692941 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.692916    2571 generic.go:356] "Generic (PLEG): container finished" podID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerID="7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e" exitCode=0
Sep 11 18:09:49.693054 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.692964    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"
Sep 11 18:09:49.693104 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.692964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" event={"ID":"0e735cbf-bcba-4c75-947a-9e15461b78b8","Type":"ContainerDied","Data":"7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e"}
Sep 11 18:09:49.693144 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.693107    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b" event={"ID":"0e735cbf-bcba-4c75-947a-9e15461b78b8","Type":"ContainerDied","Data":"fc1dfe8a4f4eeb51860806411d9ba37f9def6b27e307f72ff6601d694f3c0742"}
Sep 11 18:09:49.693184 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.693144    2571 scope.go:117] "RemoveContainer" containerID="7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e"
Sep 11 18:09:49.704060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.704044    2571 scope.go:117] "RemoveContainer" containerID="26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb"
Sep 11 18:09:49.712632 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.712617    2571 scope.go:117] "RemoveContainer" containerID="7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e"
Sep 11 18:09:49.712881 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:09:49.712862    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e\": container with ID starting with 7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e not found: ID does not exist" containerID="7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e"
Sep 11 18:09:49.712920 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.712892    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e"} err="failed to get container status \"7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e\": rpc error: code = NotFound desc = could not find container \"7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e\": container with ID starting with 7eb7ce3d730473de62616241a9dcd177b61a8644d6286999c7d8f1b8b7351f8e not found: ID does not exist"
Sep 11 18:09:49.712920 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.712909    2571 scope.go:117] "RemoveContainer" containerID="26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb"
Sep 11 18:09:49.713183 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:09:49.713160    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb\": container with ID starting with 26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb not found: ID does not exist" containerID="26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb"
Sep 11 18:09:49.713267 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.713185    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb"} err="failed to get container status \"26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb\": rpc error: code = NotFound desc = could not find container \"26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb\": container with ID starting with 26f8ffcbcda8be92b85025db94c5398a7addb12ac40338d36c734f633d9d8fdb not found: ID does not exist"
Sep 11 18:09:49.717627 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.717585    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76" podStartSLOduration=1.717574018 podStartE2EDuration="1.717574018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:49.715828264 +0000 UTC m=+953.485894611" watchObservedRunningTime="2026-09-11 18:09:49.717574018 +0000 UTC m=+953.487640366"
Sep 11 18:09:49.731879 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.731853    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"]
Sep 11 18:09:49.736533 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:49.736510    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-820e15b2-openshift-default-574cc56b46-tsx2b"]
Sep 11 18:09:50.731677 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:50.731638    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" path="/var/lib/kubelet/pods/0e735cbf-bcba-4c75-947a-9e15461b78b8/volumes"
Sep 11 18:09:55.203873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.203839    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"]
Sep 11 18:09:55.204257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.204245    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy"
Sep 11 18:09:55.204316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.204258    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy"
Sep 11 18:09:55.204316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.204267    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy"
Sep 11 18:09:55.204316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.204272    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy"
Sep 11 18:09:55.204420 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.204358    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy"
Sep 11 18:09:55.204420 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.204369    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e735cbf-bcba-4c75-947a-9e15461b78b8" containerName="istio-proxy"
Sep 11 18:09:55.207505 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.207489    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.212675 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.212653    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-93598a\"/\"test-model-93598a-kserve-self-signed-certs\""
Sep 11 18:09:55.212822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.212686    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-93598a\"/\"openshift-service-ca.crt\""
Sep 11 18:09:55.212822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.212710    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-93598a\"/\"default-dockercfg-cfvl9\""
Sep 11 18:09:55.212822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.212692    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-93598a\"/\"kube-root-ca.crt\""
Sep 11 18:09:55.217906 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.217885    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"]
Sep 11 18:09:55.346607 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.346569    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dfgz\" (UniqueName: \"kubernetes.io/projected/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-kube-api-access-4dfgz\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.346607 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.346615    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-model-cache\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.346833 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.346679    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tmp-dir\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.346833 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.346733    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-dshm\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.346833 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.346758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-home\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.346931 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.346844    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tls-certs\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.447587 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.447544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-dshm\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.447796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.447610    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-home\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.447796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.447666    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tls-certs\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.447796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.447753    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dfgz\" (UniqueName: \"kubernetes.io/projected/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-kube-api-access-4dfgz\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.447796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.447783    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-model-cache\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.448012 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.447836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tmp-dir\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.448394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.448073    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-home\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.448394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.448159    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tmp-dir\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.448394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.448354    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-model-cache\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.450627 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.450600    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-dshm\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.450746 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.450728    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tls-certs\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.457379 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.457253    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dfgz\" (UniqueName: \"kubernetes.io/projected/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-kube-api-access-4dfgz\") pod \"test-model-93598a-kserve-5c598875f-vgqrc\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") " pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.517626 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.517594    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:09:55.807828 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:55.807797    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"]
Sep 11 18:09:55.808006 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:09:55.807983    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf2f26e5_3de0_4b9c_96a9_31a61ebd146f.slice/crio-c480478c479464a4f45a96848ca8a14c6efca97da444e8d22f7b4b8861ab26d9 WatchSource:0}: Error finding container c480478c479464a4f45a96848ca8a14c6efca97da444e8d22f7b4b8861ab26d9: Status 404 returned error can't find the container with id c480478c479464a4f45a96848ca8a14c6efca97da444e8d22f7b4b8861ab26d9
Sep 11 18:09:56.740701 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:56.740667    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc" event={"ID":"df2f26e5-3de0-4b9c-96a9-31a61ebd146f","Type":"ContainerStarted","Data":"f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9"}
Sep 11 18:09:56.740701 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:56.740705    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc" event={"ID":"df2f26e5-3de0-4b9c-96a9-31a61ebd146f","Type":"ContainerStarted","Data":"c480478c479464a4f45a96848ca8a14c6efca97da444e8d22f7b4b8861ab26d9"}
Sep 11 18:09:56.775311 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:09:56.775254    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc" podStartSLOduration=1.775235597 podStartE2EDuration="1.775235597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:56.773625325 +0000 UTC m=+960.543691675" watchObservedRunningTime="2026-09-11 18:09:56.775235597 +0000 UTC m=+960.545301945"
Sep 11 18:10:00.702838 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:00.702802    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:10:05.518003 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:05.517973    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:10:05.518418 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:05.518103    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:10:05.530705 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:05.530680    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:10:05.808993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:05.808892    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:10:11.300666 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.300633    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"]
Sep 11 18:10:11.305744 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.305720    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"
Sep 11 18:10:11.308579 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.308551    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:10:11.315671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.315644    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"]
Sep 11 18:10:11.399489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.399450    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9f812-2870-4363-b512-4774aeb42a0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds\" (UID: \"dfb9f812-2870-4363-b512-4774aeb42a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"
Sep 11 18:10:11.501032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.500966    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9f812-2870-4363-b512-4774aeb42a0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds\" (UID: \"dfb9f812-2870-4363-b512-4774aeb42a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"
Sep 11 18:10:11.501669 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.501649    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9f812-2870-4363-b512-4774aeb42a0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds\" (UID: \"dfb9f812-2870-4363-b512-4774aeb42a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"
Sep 11 18:10:11.623415 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.623327    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"
Sep 11 18:10:11.897681 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:11.897653    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"]
Sep 11 18:10:11.898852 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:10:11.898825    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb9f812_2870_4363_b512_4774aeb42a0d.slice/crio-9b9e9620079fad7ff147ddc9b5763380f3240f18366105e8d367fd81534350f9 WatchSource:0}: Error finding container 9b9e9620079fad7ff147ddc9b5763380f3240f18366105e8d367fd81534350f9: Status 404 returned error can't find the container with id 9b9e9620079fad7ff147ddc9b5763380f3240f18366105e8d367fd81534350f9
Sep 11 18:10:12.841736 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:12.841654    2571 generic.go:356] "Generic (PLEG): container finished" podID="dfb9f812-2870-4363-b512-4774aeb42a0d" containerID="f689e6d5bd479bead81a30b7c70e415eec72d2d57544e2d5c3959a3def56abf4" exitCode=0
Sep 11 18:10:12.842099 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:12.841738    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds" event={"ID":"dfb9f812-2870-4363-b512-4774aeb42a0d","Type":"ContainerDied","Data":"f689e6d5bd479bead81a30b7c70e415eec72d2d57544e2d5c3959a3def56abf4"}
Sep 11 18:10:12.842099 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:12.841786    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds" event={"ID":"dfb9f812-2870-4363-b512-4774aeb42a0d","Type":"ContainerStarted","Data":"9b9e9620079fad7ff147ddc9b5763380f3240f18366105e8d367fd81534350f9"}
Sep 11 18:10:13.980227 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:13.980203    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"
Sep 11 18:10:14.128272 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:14.128172    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9f812-2870-4363-b512-4774aeb42a0d-maas-api-service-ca\") pod \"dfb9f812-2870-4363-b512-4774aeb42a0d\" (UID: \"dfb9f812-2870-4363-b512-4774aeb42a0d\") "
Sep 11 18:10:14.128537 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:14.128512    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb9f812-2870-4363-b512-4774aeb42a0d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfb9f812-2870-4363-b512-4774aeb42a0d" (UID: "dfb9f812-2870-4363-b512-4774aeb42a0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:10:14.229309 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:14.229270    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfb9f812-2870-4363-b512-4774aeb42a0d-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:14.856875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:14.856845    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"
Sep 11 18:10:14.857048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:14.856874    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds" event={"ID":"dfb9f812-2870-4363-b512-4774aeb42a0d","Type":"ContainerDied","Data":"9b9e9620079fad7ff147ddc9b5763380f3240f18366105e8d367fd81534350f9"}
Sep 11 18:10:14.857048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:14.856917    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9e9620079fad7ff147ddc9b5763380f3240f18366105e8d367fd81534350f9"
Sep 11 18:10:18.072331 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.072285    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"]
Sep 11 18:10:18.072869 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.072511    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76" podUID="9b2c16f0-27c7-4b54-a078-29552a7c2c86" containerName="payload-pre-processing" containerID="cri-o://070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa" gracePeriod=30
Sep 11 18:10:18.329966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.329883    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"]
Sep 11 18:10:18.333230 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.333204    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-85c5a226-cr8ds"]
Sep 11 18:10:18.333637 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.333620    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:10:18.367417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.367386    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wzdx\" (UniqueName: \"kubernetes.io/projected/9b2c16f0-27c7-4b54-a078-29552a7c2c86-kube-api-access-4wzdx\") pod \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\" (UID: \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\") "
Sep 11 18:10:18.367617 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.367511    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2c16f0-27c7-4b54-a078-29552a7c2c86-plugins-config-volume\") pod \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\" (UID: \"9b2c16f0-27c7-4b54-a078-29552a7c2c86\") "
Sep 11 18:10:18.368186 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.368155    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2c16f0-27c7-4b54-a078-29552a7c2c86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b2c16f0-27c7-4b54-a078-29552a7c2c86" (UID: "9b2c16f0-27c7-4b54-a078-29552a7c2c86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:10:18.370160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.370135    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2c16f0-27c7-4b54-a078-29552a7c2c86-kube-api-access-4wzdx" (OuterVolumeSpecName: "kube-api-access-4wzdx") pod "9b2c16f0-27c7-4b54-a078-29552a7c2c86" (UID: "9b2c16f0-27c7-4b54-a078-29552a7c2c86"). InnerVolumeSpecName "kube-api-access-4wzdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:18.468401 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.468365    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzdx\" (UniqueName: \"kubernetes.io/projected/9b2c16f0-27c7-4b54-a078-29552a7c2c86-kube-api-access-4wzdx\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:18.468401 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.468400    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b2c16f0-27c7-4b54-a078-29552a7c2c86-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:18.732311 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.732272    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfb9f812-2870-4363-b512-4774aeb42a0d" path="/var/lib/kubelet/pods/dfb9f812-2870-4363-b512-4774aeb42a0d/volumes"
Sep 11 18:10:18.883147 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.883113    2571 generic.go:356] "Generic (PLEG): container finished" podID="9b2c16f0-27c7-4b54-a078-29552a7c2c86" containerID="070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa" exitCode=0
Sep 11 18:10:18.883377 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.883159    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"
Sep 11 18:10:18.883377 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.883199    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76" event={"ID":"9b2c16f0-27c7-4b54-a078-29552a7c2c86","Type":"ContainerDied","Data":"070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa"}
Sep 11 18:10:18.883377 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.883269    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76" event={"ID":"9b2c16f0-27c7-4b54-a078-29552a7c2c86","Type":"ContainerDied","Data":"ffa7ace6c0fb01192ffa9805b28962da0dbeee8f999b1359b1386eaa41312050"}
Sep 11 18:10:18.883377 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.883321    2571 scope.go:117] "RemoveContainer" containerID="070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa"
Sep 11 18:10:18.893011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.892990    2571 scope.go:117] "RemoveContainer" containerID="070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa"
Sep 11 18:10:18.893362 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:10:18.893342    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa\": container with ID starting with 070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa not found: ID does not exist" containerID="070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa"
Sep 11 18:10:18.893438 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.893374    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa"} err="failed to get container status \"070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa\": rpc error: code = NotFound desc = could not find container \"070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa\": container with ID starting with 070cd26114e813f9459fd4459e6a82ef47672a417e513f3e2853b1dec0088dfa not found: ID does not exist"
Sep 11 18:10:18.910099 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.910030    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"]
Sep 11 18:10:18.917553 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:18.917525    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-85c5a226-5bcfdf7bcd-r8q76"]
Sep 11 18:10:20.730996 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:20.730965    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b2c16f0-27c7-4b54-a078-29552a7c2c86" path="/var/lib/kubelet/pods/9b2c16f0-27c7-4b54-a078-29552a7c2c86/volumes"
Sep 11 18:10:33.921312 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:33.921265    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"]
Sep 11 18:10:33.921736 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:33.921513    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy" containerID="cri-o://b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f" gracePeriod=30
Sep 11 18:10:39.178672 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.178645    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:10:39.261097 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261015    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-token\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261097 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261058    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-certs\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261097 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261095    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-envoy\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261411 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261277    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-socket\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261411 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261338    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-podinfo\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261411 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261363    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-credential-socket\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261411 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261399    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e966a2f4-4fcc-4478-93b8-13100af36dd8-istiod-ca-cert\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261411 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261399    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:39.261663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261441    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp8f6\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-kube-api-access-pp8f6\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261508    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-data\") pod \"e966a2f4-4fcc-4478-93b8-13100af36dd8\" (UID: \"e966a2f4-4fcc-4478-93b8-13100af36dd8\") "
Sep 11 18:10:39.261766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261672    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:39.262211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261923    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:39.262211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.261975    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e966a2f4-4fcc-4478-93b8-13100af36dd8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:10:39.262211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.262113    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-data" (OuterVolumeSpecName: "istio-data") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:39.262211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.262130    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:39.262211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.262151    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:39.262211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.262165    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:39.262211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.262179    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e966a2f4-4fcc-4478-93b8-13100af36dd8-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:39.263394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.263371    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-token" (OuterVolumeSpecName: "istio-token") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:39.263478 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.263457    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:39.263671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.263646    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:10:39.263871 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.263856    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-kube-api-access-pp8f6" (OuterVolumeSpecName: "kube-api-access-pp8f6") pod "e966a2f4-4fcc-4478-93b8-13100af36dd8" (UID: "e966a2f4-4fcc-4478-93b8-13100af36dd8"). InnerVolumeSpecName "kube-api-access-pp8f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:39.362763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.362725    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp8f6\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-kube-api-access-pp8f6\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:39.362763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.362752    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:39.362763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.362761    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:39.362763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.362769    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:39.362763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:39.362776    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e966a2f4-4fcc-4478-93b8-13100af36dd8-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:10:40.026146 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.026112    2571 generic.go:356] "Generic (PLEG): container finished" podID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerID="b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f" exitCode=0
Sep 11 18:10:40.026368 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.026165    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"
Sep 11 18:10:40.026368 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.026196    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" event={"ID":"e966a2f4-4fcc-4478-93b8-13100af36dd8","Type":"ContainerDied","Data":"b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f"}
Sep 11 18:10:40.026368 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.026252    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2" event={"ID":"e966a2f4-4fcc-4478-93b8-13100af36dd8","Type":"ContainerDied","Data":"c6c04f92a041bd17996fe5ead0be9e40b3dffac5a4cb0952b7f9e82aa9d4a736"}
Sep 11 18:10:40.026368 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.026273    2571 scope.go:117] "RemoveContainer" containerID="b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f"
Sep 11 18:10:40.036458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.036441    2571 scope.go:117] "RemoveContainer" containerID="9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7"
Sep 11 18:10:40.045036 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.044882    2571 scope.go:117] "RemoveContainer" containerID="b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f"
Sep 11 18:10:40.045174 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:10:40.045156    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f\": container with ID starting with b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f not found: ID does not exist" containerID="b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f"
Sep 11 18:10:40.045218 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.045184    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f"} err="failed to get container status \"b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f\": rpc error: code = NotFound desc = could not find container \"b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f\": container with ID starting with b4d6cfe8a891427901ebd25ed27d23bd8d24f77a26931c87fd5cbfc773a29c6f not found: ID does not exist"
Sep 11 18:10:40.045218 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.045201    2571 scope.go:117] "RemoveContainer" containerID="9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7"
Sep 11 18:10:40.045455 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:10:40.045437    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7\": container with ID starting with 9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7 not found: ID does not exist" containerID="9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7"
Sep 11 18:10:40.045500 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.045461    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7"} err="failed to get container status \"9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7\": rpc error: code = NotFound desc = could not find container \"9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7\": container with ID starting with 9360ddc45734bd1d398334ef89355c532720afb41454d6f780db5a095bb26ed7 not found: ID does not exist"
Sep 11 18:10:40.051231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.051208    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"]
Sep 11 18:10:40.055041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.055022    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-736ba3-openshift-default-85cbfc4798-2z7t2"]
Sep 11 18:10:40.730915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:40.730882    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" path="/var/lib/kubelet/pods/e966a2f4-4fcc-4478-93b8-13100af36dd8/volumes"
Sep 11 18:10:42.685902 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.685866    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"]
Sep 11 18:10:42.686337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686320    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy"
Sep 11 18:10:42.686393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686341    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy"
Sep 11 18:10:42.686393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686360    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b2c16f0-27c7-4b54-a078-29552a7c2c86" containerName="payload-pre-processing"
Sep 11 18:10:42.686393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686366    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b2c16f0-27c7-4b54-a078-29552a7c2c86" containerName="payload-pre-processing"
Sep 11 18:10:42.686393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686378    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb9f812-2870-4363-b512-4774aeb42a0d" containerName="revoke-keys"
Sep 11 18:10:42.686393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686383    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb9f812-2870-4363-b512-4774aeb42a0d" containerName="revoke-keys"
Sep 11 18:10:42.686552 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686457    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb9f812-2870-4363-b512-4774aeb42a0d" containerName="revoke-keys"
Sep 11 18:10:42.686552 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686466    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy"
Sep 11 18:10:42.686552 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.686473    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b2c16f0-27c7-4b54-a078-29552a7c2c86" containerName="payload-pre-processing"
Sep 11 18:10:42.689720 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.689699    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:42.692470 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.692447    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-aedf1a\""
Sep 11 18:10:42.704525 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.704496    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"]
Sep 11 18:10:42.732572 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.732542    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"]
Sep 11 18:10:42.732930 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.732918    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy"
Sep 11 18:10:42.732972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.732933    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy"
Sep 11 18:10:42.733017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.733000    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e966a2f4-4fcc-4478-93b8-13100af36dd8" containerName="istio-proxy"
Sep 11 18:10:42.735954 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.735933    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:42.738723 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.738705    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-aedf1a-dockercfg-xxvnp\""
Sep 11 18:10:42.740095 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.740072    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"]
Sep 11 18:10:42.794132 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.794096    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29698cf8-0bdc-4498-80a7-b8e8d784b47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5\" (UID: \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:42.794347 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.794179    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfgmw\" (UniqueName: \"kubernetes.io/projected/29698cf8-0bdc-4498-80a7-b8e8d784b47d-kube-api-access-kfgmw\") pod \"payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5\" (UID: \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:42.895185 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.895149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfgmw\" (UniqueName: \"kubernetes.io/projected/29698cf8-0bdc-4498-80a7-b8e8d784b47d-kube-api-access-kfgmw\") pod \"payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5\" (UID: \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:42.895394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.895198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d9d28c-2f91-4588-bb43-19b40eabb563-plugins-config-volume\") pod \"payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf\" (UID: \"a7d9d28c-2f91-4588-bb43-19b40eabb563\") " pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:42.895394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.895250    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29698cf8-0bdc-4498-80a7-b8e8d784b47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5\" (UID: \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:42.895394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.895279    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbxb\" (UniqueName: \"kubernetes.io/projected/a7d9d28c-2f91-4588-bb43-19b40eabb563-kube-api-access-chbxb\") pod \"payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf\" (UID: \"a7d9d28c-2f91-4588-bb43-19b40eabb563\") " pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:42.895980 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.895960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29698cf8-0bdc-4498-80a7-b8e8d784b47d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5\" (UID: \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:42.903962 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.903933    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfgmw\" (UniqueName: \"kubernetes.io/projected/29698cf8-0bdc-4498-80a7-b8e8d784b47d-kube-api-access-kfgmw\") pod \"payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5\" (UID: \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:42.996880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.996794    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbxb\" (UniqueName: \"kubernetes.io/projected/a7d9d28c-2f91-4588-bb43-19b40eabb563-kube-api-access-chbxb\") pod \"payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf\" (UID: \"a7d9d28c-2f91-4588-bb43-19b40eabb563\") " pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:42.996880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.996857    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d9d28c-2f91-4588-bb43-19b40eabb563-plugins-config-volume\") pod \"payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf\" (UID: \"a7d9d28c-2f91-4588-bb43-19b40eabb563\") " pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:42.997590 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:42.997570    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d9d28c-2f91-4588-bb43-19b40eabb563-plugins-config-volume\") pod \"payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf\" (UID: \"a7d9d28c-2f91-4588-bb43-19b40eabb563\") " pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:43.000658 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:43.000638    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:43.010037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:43.010013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbxb\" (UniqueName: \"kubernetes.io/projected/a7d9d28c-2f91-4588-bb43-19b40eabb563-kube-api-access-chbxb\") pod \"payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf\" (UID: \"a7d9d28c-2f91-4588-bb43-19b40eabb563\") " pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:43.049136 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:43.049023    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:43.283991 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:43.283959    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"]
Sep 11 18:10:43.284864 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:10:43.284836    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29698cf8_0bdc_4498_80a7_b8e8d784b47d.slice/crio-49cfe853eb7c585cd25245fbcb409268c4e3f8d10b94413d0ebfab3743cadc4e WatchSource:0}: Error finding container 49cfe853eb7c585cd25245fbcb409268c4e3f8d10b94413d0ebfab3743cadc4e: Status 404 returned error can't find the container with id 49cfe853eb7c585cd25245fbcb409268c4e3f8d10b94413d0ebfab3743cadc4e
Sep 11 18:10:43.341732 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:43.341707    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"]
Sep 11 18:10:43.343392 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:10:43.343369    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d9d28c_2f91_4588_bb43_19b40eabb563.slice/crio-def0dd074957b8365e7631f27ff24ebf795365a068d3a53edf258f2654ced583 WatchSource:0}: Error finding container def0dd074957b8365e7631f27ff24ebf795365a068d3a53edf258f2654ced583: Status 404 returned error can't find the container with id def0dd074957b8365e7631f27ff24ebf795365a068d3a53edf258f2654ced583
Sep 11 18:10:44.055681 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:44.055556    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf" event={"ID":"a7d9d28c-2f91-4588-bb43-19b40eabb563","Type":"ContainerStarted","Data":"f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d"}
Sep 11 18:10:44.056140 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:44.055842    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:10:44.056140 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:44.055901    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf" event={"ID":"a7d9d28c-2f91-4588-bb43-19b40eabb563","Type":"ContainerStarted","Data":"def0dd074957b8365e7631f27ff24ebf795365a068d3a53edf258f2654ced583"}
Sep 11 18:10:44.057868 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:44.057839    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5" event={"ID":"29698cf8-0bdc-4498-80a7-b8e8d784b47d","Type":"ContainerStarted","Data":"56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c"}
Sep 11 18:10:44.057996 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:44.057885    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5" event={"ID":"29698cf8-0bdc-4498-80a7-b8e8d784b47d","Type":"ContainerStarted","Data":"49cfe853eb7c585cd25245fbcb409268c4e3f8d10b94413d0ebfab3743cadc4e"}
Sep 11 18:10:44.057996 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:44.057968    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:44.075284 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:44.075237    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf" podStartSLOduration=2.0752253 podStartE2EDuration="2.0752253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:44.074645368 +0000 UTC m=+1007.844711739" watchObservedRunningTime="2026-09-11 18:10:44.0752253 +0000 UTC m=+1007.845291648"
Sep 11 18:10:44.095346 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:44.095283    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5" podStartSLOduration=2.095267176 podStartE2EDuration="2.095267176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:44.094233872 +0000 UTC m=+1007.864300220" watchObservedRunningTime="2026-09-11 18:10:44.095267176 +0000 UTC m=+1007.865333524"
Sep 11 18:10:55.067923 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:55.067887    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:10:55.068432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:10:55.067961    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:12:11.255053 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.254964    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"]
Sep 11 18:12:11.258644 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.258628    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:11.261846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.261825    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7c2684c9\""
Sep 11 18:12:11.262233 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.262216    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7c2684c9-dockercfg-tvm2v\""
Sep 11 18:12:11.269649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.269626    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"]
Sep 11 18:12:11.310790 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.310757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cb2aa5-0658-4673-83c2-839b3afa0c79-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c2684c9-769db79665-q66pp\" (UID: \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\") " pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:11.310989 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.310932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64zc\" (UniqueName: \"kubernetes.io/projected/a6cb2aa5-0658-4673-83c2-839b3afa0c79-kube-api-access-p64zc\") pod \"payload-processing-e2e-mt-7c2684c9-769db79665-q66pp\" (UID: \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\") " pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:11.412125 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.412070    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64zc\" (UniqueName: \"kubernetes.io/projected/a6cb2aa5-0658-4673-83c2-839b3afa0c79-kube-api-access-p64zc\") pod \"payload-processing-e2e-mt-7c2684c9-769db79665-q66pp\" (UID: \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\") " pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:11.412345 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.412166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cb2aa5-0658-4673-83c2-839b3afa0c79-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c2684c9-769db79665-q66pp\" (UID: \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\") " pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:11.412882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.412857    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cb2aa5-0658-4673-83c2-839b3afa0c79-plugins-config-volume\") pod \"payload-processing-e2e-mt-7c2684c9-769db79665-q66pp\" (UID: \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\") " pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:11.420941 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.420920    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64zc\" (UniqueName: \"kubernetes.io/projected/a6cb2aa5-0658-4673-83c2-839b3afa0c79-kube-api-access-p64zc\") pod \"payload-processing-e2e-mt-7c2684c9-769db79665-q66pp\" (UID: \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\") " pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:11.570328 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.570233    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:11.841627 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:11.841594    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"]
Sep 11 18:12:11.842665 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:12:11.842639    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cb2aa5_0658_4673_83c2_839b3afa0c79.slice/crio-5f076ba6d59293968f0213b1cbf1ea26fed6edc1067f53793d9fa4870daca602 WatchSource:0}: Error finding container 5f076ba6d59293968f0213b1cbf1ea26fed6edc1067f53793d9fa4870daca602: Status 404 returned error can't find the container with id 5f076ba6d59293968f0213b1cbf1ea26fed6edc1067f53793d9fa4870daca602
Sep 11 18:12:12.637080 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:12.637044    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp" event={"ID":"a6cb2aa5-0658-4673-83c2-839b3afa0c79","Type":"ContainerStarted","Data":"656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27"}
Sep 11 18:12:12.637521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:12.637208    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:12.637521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:12.637228    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp" event={"ID":"a6cb2aa5-0658-4673-83c2-839b3afa0c79","Type":"ContainerStarted","Data":"5f076ba6d59293968f0213b1cbf1ea26fed6edc1067f53793d9fa4870daca602"}
Sep 11 18:12:12.659013 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:12.658958    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp" podStartSLOduration=1.658940158 podStartE2EDuration="1.658940158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:12.657936799 +0000 UTC m=+1096.428003151" watchObservedRunningTime="2026-09-11 18:12:12.658940158 +0000 UTC m=+1096.429006577"
Sep 11 18:12:23.645914 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:23.645877    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:39.931531 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.931502    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"]
Sep 11 18:12:39.935320 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.935290    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:39.939417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.939393    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-aedf1a\"/\"kube-root-ca.crt\""
Sep 11 18:12:39.939551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.939419    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-aedf1a\"/\"openshift-service-ca.crt\""
Sep 11 18:12:39.939551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.939397    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-aedf1a\"/\"default-dockercfg-vd2pc\""
Sep 11 18:12:39.939551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.939393    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-aedf1a\"/\"explicit-model-aedf1a-kserve-self-signed-certs\""
Sep 11 18:12:39.947076 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.947051    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"]
Sep 11 18:12:39.984621 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.984596    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tmp-dir\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:39.984798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.984642    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl8z6\" (UniqueName: \"kubernetes.io/projected/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-kube-api-access-cl8z6\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:39.984798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.984660    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-home\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:39.984929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.984788    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tls-certs\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:39.984929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.984826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-model-cache\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:39.984929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:39.984897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-dshm\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.085982 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.085942    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tmp-dir\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.086176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.086000    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8z6\" (UniqueName: \"kubernetes.io/projected/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-kube-api-access-cl8z6\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.086176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.086019    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-home\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.086176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.086071    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tls-certs\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.086391 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.086186    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-model-cache\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.086391 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.086248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-dshm\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.086391 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.086372    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tmp-dir\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.086511 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.086414    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-home\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.086558 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.086527    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-model-cache\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.088462 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.088433    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-dshm\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.088697 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.088679    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tls-certs\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.095447 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.095418    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8z6\" (UniqueName: \"kubernetes.io/projected/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-kube-api-access-cl8z6\") pod \"explicit-model-aedf1a-kserve-64f9b5889-dbdj2\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") " pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.244848 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.244766    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:40.525734 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.525703    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"]
Sep 11 18:12:40.526220 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:12:40.526196    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f11aa4d_2037_4e3c_9cd8_c2995de8b84d.slice/crio-2689ba4526e1590424d98153fa4dfe73c792a934dbda8a32ad8bcd0d678fb0d7 WatchSource:0}: Error finding container 2689ba4526e1590424d98153fa4dfe73c792a934dbda8a32ad8bcd0d678fb0d7: Status 404 returned error can't find the container with id 2689ba4526e1590424d98153fa4dfe73c792a934dbda8a32ad8bcd0d678fb0d7
Sep 11 18:12:40.821537 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.821451    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2" event={"ID":"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d","Type":"ContainerStarted","Data":"120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf"}
Sep 11 18:12:40.821537 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.821493    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2" event={"ID":"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d","Type":"ContainerStarted","Data":"2689ba4526e1590424d98153fa4dfe73c792a934dbda8a32ad8bcd0d678fb0d7"}
Sep 11 18:12:40.841233 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:40.841187    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2" podStartSLOduration=1.8411687429999999 podStartE2EDuration="1.841168743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:40.840414308 +0000 UTC m=+1124.610480656" watchObservedRunningTime="2026-09-11 18:12:40.841168743 +0000 UTC m=+1124.611235090"
Sep 11 18:12:44.776983 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:44.776943    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"]
Sep 11 18:12:44.781017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:44.780988    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"
Sep 11 18:12:44.784008 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:44.783984    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:12:44.789203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:44.789180    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"]
Sep 11 18:12:44.830368 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:44.830332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8daa4-b172-41d7-9d11-f81cefceaa88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz\" (UID: \"c6f8daa4-b172-41d7-9d11-f81cefceaa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"
Sep 11 18:12:44.931137 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:44.931102    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8daa4-b172-41d7-9d11-f81cefceaa88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz\" (UID: \"c6f8daa4-b172-41d7-9d11-f81cefceaa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"
Sep 11 18:12:44.931783 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:44.931761    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8daa4-b172-41d7-9d11-f81cefceaa88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz\" (UID: \"c6f8daa4-b172-41d7-9d11-f81cefceaa88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"
Sep 11 18:12:45.094597 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:45.094510    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"
Sep 11 18:12:45.371745 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:45.371702    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"]
Sep 11 18:12:45.372925 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:12:45.372900    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f8daa4_b172_41d7_9d11_f81cefceaa88.slice/crio-a7d118511e8e8a95b9cc615e0d559c96d58339a9cd203c771b5eeff4da1d4f4c WatchSource:0}: Error finding container a7d118511e8e8a95b9cc615e0d559c96d58339a9cd203c771b5eeff4da1d4f4c: Status 404 returned error can't find the container with id a7d118511e8e8a95b9cc615e0d559c96d58339a9cd203c771b5eeff4da1d4f4c
Sep 11 18:12:45.855652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:45.855616    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz" event={"ID":"c6f8daa4-b172-41d7-9d11-f81cefceaa88","Type":"ContainerStarted","Data":"85a313c685c89dd8ed046fd8a05f7a553ee57998607181d5a3e970d1374b113f"}
Sep 11 18:12:45.855652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:45.855668    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz" event={"ID":"c6f8daa4-b172-41d7-9d11-f81cefceaa88","Type":"ContainerStarted","Data":"a7d118511e8e8a95b9cc615e0d559c96d58339a9cd203c771b5eeff4da1d4f4c"}
Sep 11 18:12:45.873431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:45.873384    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz" podStartSLOduration=1.873368847 podStartE2EDuration="1.873368847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:45.872702487 +0000 UTC m=+1129.642768835" watchObservedRunningTime="2026-09-11 18:12:45.873368847 +0000 UTC m=+1129.643435195"
Sep 11 18:12:46.864238 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:46.864204    2571 generic.go:356] "Generic (PLEG): container finished" podID="c6f8daa4-b172-41d7-9d11-f81cefceaa88" containerID="85a313c685c89dd8ed046fd8a05f7a553ee57998607181d5a3e970d1374b113f" exitCode=0
Sep 11 18:12:46.864652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:46.864273    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz" event={"ID":"c6f8daa4-b172-41d7-9d11-f81cefceaa88","Type":"ContainerDied","Data":"85a313c685c89dd8ed046fd8a05f7a553ee57998607181d5a3e970d1374b113f"}
Sep 11 18:12:48.024840 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:48.024810    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"
Sep 11 18:12:48.062165 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:48.062065    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8daa4-b172-41d7-9d11-f81cefceaa88-maas-api-service-ca\") pod \"c6f8daa4-b172-41d7-9d11-f81cefceaa88\" (UID: \"c6f8daa4-b172-41d7-9d11-f81cefceaa88\") "
Sep 11 18:12:48.062705 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:48.062677    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f8daa4-b172-41d7-9d11-f81cefceaa88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6f8daa4-b172-41d7-9d11-f81cefceaa88" (UID: "c6f8daa4-b172-41d7-9d11-f81cefceaa88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:12:48.163793 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:48.163752    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8daa4-b172-41d7-9d11-f81cefceaa88-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:12:48.885209 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:48.885177    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz" event={"ID":"c6f8daa4-b172-41d7-9d11-f81cefceaa88","Type":"ContainerDied","Data":"a7d118511e8e8a95b9cc615e0d559c96d58339a9cd203c771b5eeff4da1d4f4c"}
Sep 11 18:12:48.885209 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:48.885204    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"
Sep 11 18:12:48.885434 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:48.885226    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7d118511e8e8a95b9cc615e0d559c96d58339a9cd203c771b5eeff4da1d4f4c"
Sep 11 18:12:50.245108 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:50.245064    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:50.245108 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:50.245113    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:50.258067 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:50.258039    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:50.908197 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:50.908169    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:12:55.575788 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.575641    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"]
Sep 11 18:12:55.576321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.575907    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp" podUID="a6cb2aa5-0658-4673-83c2-839b3afa0c79" containerName="payload-processing" containerID="cri-o://656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27" gracePeriod=30
Sep 11 18:12:55.758885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.758839    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"]
Sep 11 18:12:55.761657 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.761586    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c2684c9-jxhqz"]
Sep 11 18:12:55.852632 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.852606    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:55.933365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.933324    2571 generic.go:356] "Generic (PLEG): container finished" podID="a6cb2aa5-0658-4673-83c2-839b3afa0c79" containerID="656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27" exitCode=0
Sep 11 18:12:55.933575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.933520    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp" event={"ID":"a6cb2aa5-0658-4673-83c2-839b3afa0c79","Type":"ContainerDied","Data":"656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27"}
Sep 11 18:12:55.933649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.933580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp" event={"ID":"a6cb2aa5-0658-4673-83c2-839b3afa0c79","Type":"ContainerDied","Data":"5f076ba6d59293968f0213b1cbf1ea26fed6edc1067f53793d9fa4870daca602"}
Sep 11 18:12:55.933649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.933610    2571 scope.go:117] "RemoveContainer" containerID="656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27"
Sep 11 18:12:55.933883 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.933855    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"
Sep 11 18:12:55.940937 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.940903    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cb2aa5-0658-4673-83c2-839b3afa0c79-plugins-config-volume\") pod \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\" (UID: \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\") "
Sep 11 18:12:55.941068 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.941052    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p64zc\" (UniqueName: \"kubernetes.io/projected/a6cb2aa5-0658-4673-83c2-839b3afa0c79-kube-api-access-p64zc\") pod \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\" (UID: \"a6cb2aa5-0658-4673-83c2-839b3afa0c79\") "
Sep 11 18:12:55.941326 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.941289    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6cb2aa5-0658-4673-83c2-839b3afa0c79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6cb2aa5-0658-4673-83c2-839b3afa0c79" (UID: "a6cb2aa5-0658-4673-83c2-839b3afa0c79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:12:55.941415 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.941399    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6cb2aa5-0658-4673-83c2-839b3afa0c79-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:12:55.943088 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.943065    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6cb2aa5-0658-4673-83c2-839b3afa0c79-kube-api-access-p64zc" (OuterVolumeSpecName: "kube-api-access-p64zc") pod "a6cb2aa5-0658-4673-83c2-839b3afa0c79" (UID: "a6cb2aa5-0658-4673-83c2-839b3afa0c79"). InnerVolumeSpecName "kube-api-access-p64zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:12:55.945507 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.945482    2571 scope.go:117] "RemoveContainer" containerID="656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27"
Sep 11 18:12:55.945831 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:12:55.945811    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27\": container with ID starting with 656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27 not found: ID does not exist" containerID="656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27"
Sep 11 18:12:55.945885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:55.945841    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27"} err="failed to get container status \"656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27\": rpc error: code = NotFound desc = could not find container \"656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27\": container with ID starting with 656379d35a14a4bdc5dd88c75a2001b451806592c0206ec6f76dae53e98eeb27 not found: ID does not exist"
Sep 11 18:12:56.042809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:56.042771    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64zc\" (UniqueName: \"kubernetes.io/projected/a6cb2aa5-0658-4673-83c2-839b3afa0c79-kube-api-access-p64zc\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:12:56.267191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:56.267157    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"]
Sep 11 18:12:56.269732 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:56.269707    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7c2684c9-769db79665-q66pp"]
Sep 11 18:12:56.731744 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:56.731714    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6cb2aa5-0658-4673-83c2-839b3afa0c79" path="/var/lib/kubelet/pods/a6cb2aa5-0658-4673-83c2-839b3afa0c79/volumes"
Sep 11 18:12:56.732103 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:12:56.732045    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f8daa4-b172-41d7-9d11-f81cefceaa88" path="/var/lib/kubelet/pods/c6f8daa4-b172-41d7-9d11-f81cefceaa88/volumes"
Sep 11 18:13:11.478015 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.477981    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"]
Sep 11 18:13:11.478462 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.478448    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6cb2aa5-0658-4673-83c2-839b3afa0c79" containerName="payload-processing"
Sep 11 18:13:11.478520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.478463    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6cb2aa5-0658-4673-83c2-839b3afa0c79" containerName="payload-processing"
Sep 11 18:13:11.478520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.478477    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f8daa4-b172-41d7-9d11-f81cefceaa88" containerName="revoke-keys"
Sep 11 18:13:11.478520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.478482    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f8daa4-b172-41d7-9d11-f81cefceaa88" containerName="revoke-keys"
Sep 11 18:13:11.478663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.478563    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f8daa4-b172-41d7-9d11-f81cefceaa88" containerName="revoke-keys"
Sep 11 18:13:11.478663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.478571    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6cb2aa5-0658-4673-83c2-839b3afa0c79" containerName="payload-processing"
Sep 11 18:13:11.483254 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.483231    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"
Sep 11 18:13:11.486046 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.486027    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:13:11.493380 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.493358    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"]
Sep 11 18:13:11.603565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.603526    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9db3124-ba6b-4c88-9586-e3397ae4c76a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks\" (UID: \"e9db3124-ba6b-4c88-9586-e3397ae4c76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"
Sep 11 18:13:11.704426 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.704388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9db3124-ba6b-4c88-9586-e3397ae4c76a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks\" (UID: \"e9db3124-ba6b-4c88-9586-e3397ae4c76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"
Sep 11 18:13:11.705027 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.705006    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9db3124-ba6b-4c88-9586-e3397ae4c76a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks\" (UID: \"e9db3124-ba6b-4c88-9586-e3397ae4c76a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"
Sep 11 18:13:11.795049 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:11.794963    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"
Sep 11 18:13:12.077564 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:12.077535    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"]
Sep 11 18:13:12.078991 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:13:12.078960    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9db3124_ba6b_4c88_9586_e3397ae4c76a.slice/crio-a064365a334d399410b0dd00b00af0e465ce92ca1b9ef1af80c0d8aadbe98634 WatchSource:0}: Error finding container a064365a334d399410b0dd00b00af0e465ce92ca1b9ef1af80c0d8aadbe98634: Status 404 returned error can't find the container with id a064365a334d399410b0dd00b00af0e465ce92ca1b9ef1af80c0d8aadbe98634
Sep 11 18:13:12.716811 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:12.716779    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"]
Sep 11 18:13:13.054345 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:13.054242    2571 generic.go:356] "Generic (PLEG): container finished" podID="e9db3124-ba6b-4c88-9586-e3397ae4c76a" containerID="f1e79c7e1cae7f5d18740ea729d1483c0f1952a9bbd39e3fd94179992c994ddf" exitCode=0
Sep 11 18:13:13.054345 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:13.054325    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks" event={"ID":"e9db3124-ba6b-4c88-9586-e3397ae4c76a","Type":"ContainerDied","Data":"f1e79c7e1cae7f5d18740ea729d1483c0f1952a9bbd39e3fd94179992c994ddf"}
Sep 11 18:13:13.054586 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:13.054371    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks" event={"ID":"e9db3124-ba6b-4c88-9586-e3397ae4c76a","Type":"ContainerStarted","Data":"a064365a334d399410b0dd00b00af0e465ce92ca1b9ef1af80c0d8aadbe98634"}
Sep 11 18:13:14.212248 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:14.212224    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"
Sep 11 18:13:14.227772 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:14.227752    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9db3124-ba6b-4c88-9586-e3397ae4c76a-maas-api-service-ca\") pod \"e9db3124-ba6b-4c88-9586-e3397ae4c76a\" (UID: \"e9db3124-ba6b-4c88-9586-e3397ae4c76a\") "
Sep 11 18:13:14.228098 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:14.228074    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9db3124-ba6b-4c88-9586-e3397ae4c76a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9db3124-ba6b-4c88-9586-e3397ae4c76a" (UID: "e9db3124-ba6b-4c88-9586-e3397ae4c76a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:13:14.328359 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:14.328275    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9db3124-ba6b-4c88-9586-e3397ae4c76a-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:15.069491 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:15.069417    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"
Sep 11 18:13:15.069491 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:15.069452    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks" event={"ID":"e9db3124-ba6b-4c88-9586-e3397ae4c76a","Type":"ContainerDied","Data":"a064365a334d399410b0dd00b00af0e465ce92ca1b9ef1af80c0d8aadbe98634"}
Sep 11 18:13:15.069691 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:15.069500    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a064365a334d399410b0dd00b00af0e465ce92ca1b9ef1af80c0d8aadbe98634"
Sep 11 18:13:25.248489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.248406    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"]
Sep 11 18:13:25.249413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.249381    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf" podUID="a7d9d28c-2f91-4588-bb43-19b40eabb563" containerName="payload-processing" containerID="cri-o://f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d" gracePeriod=30
Sep 11 18:13:25.261252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.261219    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"]
Sep 11 18:13:25.261725 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.261651    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5" podUID="29698cf8-0bdc-4498-80a7-b8e8d784b47d" containerName="payload-pre-processing" containerID="cri-o://56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c" gracePeriod=30
Sep 11 18:13:25.426680 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.426646    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"]
Sep 11 18:13:25.429716 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.429565    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-aedf1a-2lhks"]
Sep 11 18:13:25.537675 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.537651    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:13:25.573028 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.572992    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:13:25.631389 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.631292    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d9d28c-2f91-4588-bb43-19b40eabb563-plugins-config-volume\") pod \"a7d9d28c-2f91-4588-bb43-19b40eabb563\" (UID: \"a7d9d28c-2f91-4588-bb43-19b40eabb563\") "
Sep 11 18:13:25.631580 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.631452    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chbxb\" (UniqueName: \"kubernetes.io/projected/a7d9d28c-2f91-4588-bb43-19b40eabb563-kube-api-access-chbxb\") pod \"a7d9d28c-2f91-4588-bb43-19b40eabb563\" (UID: \"a7d9d28c-2f91-4588-bb43-19b40eabb563\") "
Sep 11 18:13:25.631580 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.631489    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfgmw\" (UniqueName: \"kubernetes.io/projected/29698cf8-0bdc-4498-80a7-b8e8d784b47d-kube-api-access-kfgmw\") pod \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\" (UID: \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\") "
Sep 11 18:13:25.631711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.631627    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29698cf8-0bdc-4498-80a7-b8e8d784b47d-plugins-config-volume\") pod \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\" (UID: \"29698cf8-0bdc-4498-80a7-b8e8d784b47d\") "
Sep 11 18:13:25.631711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.631642    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d9d28c-2f91-4588-bb43-19b40eabb563-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7d9d28c-2f91-4588-bb43-19b40eabb563" (UID: "a7d9d28c-2f91-4588-bb43-19b40eabb563"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:13:25.632014 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.631988    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29698cf8-0bdc-4498-80a7-b8e8d784b47d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29698cf8-0bdc-4498-80a7-b8e8d784b47d" (UID: "29698cf8-0bdc-4498-80a7-b8e8d784b47d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:13:25.632014 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.632005    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d9d28c-2f91-4588-bb43-19b40eabb563-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:25.633614 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.633594    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29698cf8-0bdc-4498-80a7-b8e8d784b47d-kube-api-access-kfgmw" (OuterVolumeSpecName: "kube-api-access-kfgmw") pod "29698cf8-0bdc-4498-80a7-b8e8d784b47d" (UID: "29698cf8-0bdc-4498-80a7-b8e8d784b47d"). InnerVolumeSpecName "kube-api-access-kfgmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:13:25.633683 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.633622    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d9d28c-2f91-4588-bb43-19b40eabb563-kube-api-access-chbxb" (OuterVolumeSpecName: "kube-api-access-chbxb") pod "a7d9d28c-2f91-4588-bb43-19b40eabb563" (UID: "a7d9d28c-2f91-4588-bb43-19b40eabb563"). InnerVolumeSpecName "kube-api-access-chbxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:13:25.732749 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.732716    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chbxb\" (UniqueName: \"kubernetes.io/projected/a7d9d28c-2f91-4588-bb43-19b40eabb563-kube-api-access-chbxb\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:25.732749 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.732749    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfgmw\" (UniqueName: \"kubernetes.io/projected/29698cf8-0bdc-4498-80a7-b8e8d784b47d-kube-api-access-kfgmw\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:25.732749 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:25.732760    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29698cf8-0bdc-4498-80a7-b8e8d784b47d-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:26.143261 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.143222    2571 generic.go:356] "Generic (PLEG): container finished" podID="29698cf8-0bdc-4498-80a7-b8e8d784b47d" containerID="56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c" exitCode=0
Sep 11 18:13:26.143449 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.143275    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"
Sep 11 18:13:26.143449 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.143326    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5" event={"ID":"29698cf8-0bdc-4498-80a7-b8e8d784b47d","Type":"ContainerDied","Data":"56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c"}
Sep 11 18:13:26.143449 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.143396    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5" event={"ID":"29698cf8-0bdc-4498-80a7-b8e8d784b47d","Type":"ContainerDied","Data":"49cfe853eb7c585cd25245fbcb409268c4e3f8d10b94413d0ebfab3743cadc4e"}
Sep 11 18:13:26.143449 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.143428    2571 scope.go:117] "RemoveContainer" containerID="56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c"
Sep 11 18:13:26.145243 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.145219    2571 generic.go:356] "Generic (PLEG): container finished" podID="a7d9d28c-2f91-4588-bb43-19b40eabb563" containerID="f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d" exitCode=0
Sep 11 18:13:26.145396 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.145265    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"
Sep 11 18:13:26.145396 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.145319    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf" event={"ID":"a7d9d28c-2f91-4588-bb43-19b40eabb563","Type":"ContainerDied","Data":"f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d"}
Sep 11 18:13:26.145396 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.145366    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf" event={"ID":"a7d9d28c-2f91-4588-bb43-19b40eabb563","Type":"ContainerDied","Data":"def0dd074957b8365e7631f27ff24ebf795365a068d3a53edf258f2654ced583"}
Sep 11 18:13:26.155831 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.155805    2571 scope.go:117] "RemoveContainer" containerID="56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c"
Sep 11 18:13:26.156155 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:13:26.156132    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c\": container with ID starting with 56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c not found: ID does not exist" containerID="56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c"
Sep 11 18:13:26.156235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.156162    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c"} err="failed to get container status \"56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c\": rpc error: code = NotFound desc = could not find container \"56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c\": container with ID starting with 56edd27ba6f2e89bc05f6ccf48112d73b851b81fc97ac065bac766faa724c66c not found: ID does not exist"
Sep 11 18:13:26.156235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.156177    2571 scope.go:117] "RemoveContainer" containerID="f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d"
Sep 11 18:13:26.165101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.165073    2571 scope.go:117] "RemoveContainer" containerID="f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d"
Sep 11 18:13:26.165387 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:13:26.165364    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d\": container with ID starting with f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d not found: ID does not exist" containerID="f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d"
Sep 11 18:13:26.165447 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.165395    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d"} err="failed to get container status \"f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d\": rpc error: code = NotFound desc = could not find container \"f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d\": container with ID starting with f33a93ccaabc2fc0d43c12b2b9a34b2fea647a8e3265df6ed04f131a6b377d1d not found: ID does not exist"
Sep 11 18:13:26.176565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.176538    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"]
Sep 11 18:13:26.183678 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.183621    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aedf1a-7b5b4ffd7c-sp5hf"]
Sep 11 18:13:26.199775 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.199752    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"]
Sep 11 18:13:26.206097 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.206077    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aedf1a-659977fd7d-t2dh5"]
Sep 11 18:13:26.276573 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.276547    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"]
Sep 11 18:13:26.277138 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277118    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9db3124-ba6b-4c88-9586-e3397ae4c76a" containerName="revoke-keys"
Sep 11 18:13:26.277235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277139    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9db3124-ba6b-4c88-9586-e3397ae4c76a" containerName="revoke-keys"
Sep 11 18:13:26.277235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277177    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29698cf8-0bdc-4498-80a7-b8e8d784b47d" containerName="payload-pre-processing"
Sep 11 18:13:26.277235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277187    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="29698cf8-0bdc-4498-80a7-b8e8d784b47d" containerName="payload-pre-processing"
Sep 11 18:13:26.277235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277198    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d9d28c-2f91-4588-bb43-19b40eabb563" containerName="payload-processing"
Sep 11 18:13:26.277235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277208    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d9d28c-2f91-4588-bb43-19b40eabb563" containerName="payload-processing"
Sep 11 18:13:26.277501 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277315    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d9d28c-2f91-4588-bb43-19b40eabb563" containerName="payload-processing"
Sep 11 18:13:26.277501 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277328    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9db3124-ba6b-4c88-9586-e3397ae4c76a" containerName="revoke-keys"
Sep 11 18:13:26.277501 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.277340    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="29698cf8-0bdc-4498-80a7-b8e8d784b47d" containerName="payload-pre-processing"
Sep 11 18:13:26.281813 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.281795    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:26.286855 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.286686    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b05610\""
Sep 11 18:13:26.291512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.291492    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"]
Sep 11 18:13:26.300209 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.300187    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"]
Sep 11 18:13:26.303641 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.303623    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:26.308542 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.308524    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b05610-dockercfg-bkq9q\""
Sep 11 18:13:26.318203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.318177    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"]
Sep 11 18:13:26.338686 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.338657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nvcb\" (UniqueName: \"kubernetes.io/projected/24dc2906-ff7e-47f3-ad65-a8b1e984296b-kube-api-access-9nvcb\") pod \"payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz\" (UID: \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:26.338807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.338705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k\" (UID: \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:26.338807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.338748    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k99jh\" (UniqueName: \"kubernetes.io/projected/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-kube-api-access-k99jh\") pod \"payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k\" (UID: \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:26.338912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.338823    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dc2906-ff7e-47f3-ad65-a8b1e984296b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz\" (UID: \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:26.439898 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.439865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nvcb\" (UniqueName: \"kubernetes.io/projected/24dc2906-ff7e-47f3-ad65-a8b1e984296b-kube-api-access-9nvcb\") pod \"payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz\" (UID: \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:26.440092 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.439910    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k\" (UID: \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:26.440092 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.439939    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k99jh\" (UniqueName: \"kubernetes.io/projected/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-kube-api-access-k99jh\") pod \"payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k\" (UID: \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:26.440092 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.439983    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dc2906-ff7e-47f3-ad65-a8b1e984296b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz\" (UID: \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:26.440704 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.440684    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dc2906-ff7e-47f3-ad65-a8b1e984296b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz\" (UID: \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:26.440762 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.440721    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k\" (UID: \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:26.448765 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.448703    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nvcb\" (UniqueName: \"kubernetes.io/projected/24dc2906-ff7e-47f3-ad65-a8b1e984296b-kube-api-access-9nvcb\") pod \"payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz\" (UID: \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:26.448765 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.448731    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k99jh\" (UniqueName: \"kubernetes.io/projected/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-kube-api-access-k99jh\") pod \"payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k\" (UID: \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:26.595364 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.595329    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:26.615794 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.615767    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:26.733490 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.733452    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29698cf8-0bdc-4498-80a7-b8e8d784b47d" path="/var/lib/kubelet/pods/29698cf8-0bdc-4498-80a7-b8e8d784b47d/volumes"
Sep 11 18:13:26.733843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.733828    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d9d28c-2f91-4588-bb43-19b40eabb563" path="/var/lib/kubelet/pods/a7d9d28c-2f91-4588-bb43-19b40eabb563/volumes"
Sep 11 18:13:26.734110 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.734098    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9db3124-ba6b-4c88-9586-e3397ae4c76a" path="/var/lib/kubelet/pods/e9db3124-ba6b-4c88-9586-e3397ae4c76a/volumes"
Sep 11 18:13:26.872470 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.872212    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"]
Sep 11 18:13:26.875353 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:13:26.875328    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f710b2_8a94_4c79_bf2b_6c0553fe6169.slice/crio-a1123c60cc0ff813839b5f31655507a0e8ef10fc1246bf0847231f80141275c5 WatchSource:0}: Error finding container a1123c60cc0ff813839b5f31655507a0e8ef10fc1246bf0847231f80141275c5: Status 404 returned error can't find the container with id a1123c60cc0ff813839b5f31655507a0e8ef10fc1246bf0847231f80141275c5
Sep 11 18:13:26.904067 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:26.904045    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"]
Sep 11 18:13:26.907494 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:13:26.907471    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dc2906_ff7e_47f3_ad65_a8b1e984296b.slice/crio-e3122099719d98d57527f0eecec5de19ac5a523b74aa85614bbaba8fb84b2dc9 WatchSource:0}: Error finding container e3122099719d98d57527f0eecec5de19ac5a523b74aa85614bbaba8fb84b2dc9: Status 404 returned error can't find the container with id e3122099719d98d57527f0eecec5de19ac5a523b74aa85614bbaba8fb84b2dc9
Sep 11 18:13:27.156664 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.156489    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k" event={"ID":"c5f710b2-8a94-4c79-bf2b-6c0553fe6169","Type":"ContainerStarted","Data":"36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17"}
Sep 11 18:13:27.156664 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.156598    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k" event={"ID":"c5f710b2-8a94-4c79-bf2b-6c0553fe6169","Type":"ContainerStarted","Data":"a1123c60cc0ff813839b5f31655507a0e8ef10fc1246bf0847231f80141275c5"}
Sep 11 18:13:27.156664 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.156631    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:27.159056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.159006    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz" event={"ID":"24dc2906-ff7e-47f3-ad65-a8b1e984296b","Type":"ContainerStarted","Data":"a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2"}
Sep 11 18:13:27.159056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.159056    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz" event={"ID":"24dc2906-ff7e-47f3-ad65-a8b1e984296b","Type":"ContainerStarted","Data":"e3122099719d98d57527f0eecec5de19ac5a523b74aa85614bbaba8fb84b2dc9"}
Sep 11 18:13:27.159269 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.159079    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:27.177241 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.177188    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k" podStartSLOduration=1.17717261 podStartE2EDuration="1.17717261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:27.175758194 +0000 UTC m=+1170.945824570" watchObservedRunningTime="2026-09-11 18:13:27.17717261 +0000 UTC m=+1170.947238957"
Sep 11 18:13:27.194843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.194746    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz" podStartSLOduration=1.194727722 podStartE2EDuration="1.194727722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:27.192821492 +0000 UTC m=+1170.962887840" watchObservedRunningTime="2026-09-11 18:13:27.194727722 +0000 UTC m=+1170.964794075"
Sep 11 18:13:27.737275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.737228    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2" podUID="8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" containerName="main" containerID="cri-o://120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf" gracePeriod=45
Sep 11 18:13:27.994021 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:27.993959    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:13:28.055846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.055805    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-home\") pod \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") "
Sep 11 18:13:28.055846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.055844    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl8z6\" (UniqueName: \"kubernetes.io/projected/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-kube-api-access-cl8z6\") pod \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") "
Sep 11 18:13:28.056098 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.055862    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-dshm\") pod \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") "
Sep 11 18:13:28.056098 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.055923    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-model-cache\") pod \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") "
Sep 11 18:13:28.056098 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.055945    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tls-certs\") pod \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") "
Sep 11 18:13:28.056098 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.055981    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tmp-dir\") pod \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\" (UID: \"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d\") "
Sep 11 18:13:28.056322 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.056125    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-home" (OuterVolumeSpecName: "home") pod "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" (UID: "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:28.056322 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.056287    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" (UID: "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:28.056431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.056350    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.056431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.056415    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" (UID: "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:28.058256 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.058233    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-dshm" (OuterVolumeSpecName: "dshm") pod "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" (UID: "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:13:28.058386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.058357    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-kube-api-access-cl8z6" (OuterVolumeSpecName: "kube-api-access-cl8z6") pod "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" (UID: "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d"). InnerVolumeSpecName "kube-api-access-cl8z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:13:28.058474 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.058456    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" (UID: "8f11aa4d-2037-4e3c-9cd8-c2995de8b84d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:13:28.157284 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.157242    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl8z6\" (UniqueName: \"kubernetes.io/projected/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-kube-api-access-cl8z6\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.157284 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.157278    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.157284 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.157293    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.157578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.157318    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.157578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.157328    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:13:28.170432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.170396    2571 generic.go:356] "Generic (PLEG): container finished" podID="8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" containerID="120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf" exitCode=0
Sep 11 18:13:28.170646 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.170453    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"
Sep 11 18:13:28.170646 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.170473    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2" event={"ID":"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d","Type":"ContainerDied","Data":"120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf"}
Sep 11 18:13:28.170646 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.170521    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2" event={"ID":"8f11aa4d-2037-4e3c-9cd8-c2995de8b84d","Type":"ContainerDied","Data":"2689ba4526e1590424d98153fa4dfe73c792a934dbda8a32ad8bcd0d678fb0d7"}
Sep 11 18:13:28.170646 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.170542    2571 scope.go:117] "RemoveContainer" containerID="120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf"
Sep 11 18:13:28.181229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.181206    2571 scope.go:117] "RemoveContainer" containerID="120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf"
Sep 11 18:13:28.181564 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:13:28.181541    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf\": container with ID starting with 120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf not found: ID does not exist" containerID="120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf"
Sep 11 18:13:28.181623 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.181576    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf"} err="failed to get container status \"120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf\": rpc error: code = NotFound desc = could not find container \"120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf\": container with ID starting with 120d76079052f292db9c917985cb4c96da685a4def7558f051e4570c05c804cf not found: ID does not exist"
Sep 11 18:13:28.197153 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.197125    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"]
Sep 11 18:13:28.202430 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.202404    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-aedf1a/explicit-model-aedf1a-kserve-64f9b5889-dbdj2"]
Sep 11 18:13:28.731498 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:28.731465    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" path="/var/lib/kubelet/pods/8f11aa4d-2037-4e3c-9cd8-c2995de8b84d/volumes"
Sep 11 18:13:33.574796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.574763    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"]
Sep 11 18:13:33.575255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.575200    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" containerName="main"
Sep 11 18:13:33.575255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.575213    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" containerName="main"
Sep 11 18:13:33.575398 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.575335    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f11aa4d-2037-4e3c-9cd8-c2995de8b84d" containerName="main"
Sep 11 18:13:33.578525 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.578506    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.582058 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.582039    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-cd668b-openshift-default-dockercfg-f4578\""
Sep 11 18:13:33.605316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.596665    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"]
Sep 11 18:13:33.720146 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720111    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-token\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.720146 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-envoy\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.720431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720184    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-podinfo\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.720431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720200    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-data\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.720431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720217    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-certs\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.720431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-credential-socket\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.720431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720259    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-socket\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.720431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720372    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4bl\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-kube-api-access-mx4bl\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.720648 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.720433    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049ad97c-9c3e-474b-a7bb-bc47605b6964-istiod-ca-cert\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821338 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-token\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821524 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821354    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-envoy\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821524 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-podinfo\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821524 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821425    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-data\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821524 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821489    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-certs\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821524 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821517    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-credential-socket\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821901 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821554    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-socket\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821901 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4bl\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-kube-api-access-mx4bl\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821901 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821710    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049ad97c-9c3e-474b-a7bb-bc47605b6964-istiod-ca-cert\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.821901 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821859    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-data\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.822116 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.821957    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-certs\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.822223 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.822199    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-credential-socket\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.822377 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.822293    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-socket\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.822739 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.822719    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049ad97c-9c3e-474b-a7bb-bc47605b6964-istiod-ca-cert\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.823772 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.823750    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-envoy\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.824015 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.823987    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-podinfo\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.830160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.830105    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-token\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.830344 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.830323    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4bl\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-kube-api-access-mx4bl\") pod \"e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") " pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:33.889685 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:33.889650    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:34.166019 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:34.165988    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"]
Sep 11 18:13:34.167484 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:13:34.167459    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049ad97c_9c3e_474b_a7bb_bc47605b6964.slice/crio-aeb088dc968990f41cd0deab3e99419eafff8d0ad90c15dd96df8467b518f436 WatchSource:0}: Error finding container aeb088dc968990f41cd0deab3e99419eafff8d0ad90c15dd96df8467b518f436: Status 404 returned error can't find the container with id aeb088dc968990f41cd0deab3e99419eafff8d0ad90c15dd96df8467b518f436
Sep 11 18:13:34.170204 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:34.170175    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:13:34.170270 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:34.170238    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:13:34.170270 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:34.170267    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:13:34.212242 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:34.212218    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w" event={"ID":"049ad97c-9c3e-474b-a7bb-bc47605b6964","Type":"ContainerStarted","Data":"aeb088dc968990f41cd0deab3e99419eafff8d0ad90c15dd96df8467b518f436"}
Sep 11 18:13:35.220786 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:35.220742    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w" event={"ID":"049ad97c-9c3e-474b-a7bb-bc47605b6964","Type":"ContainerStarted","Data":"13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f"}
Sep 11 18:13:35.246663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:35.246617    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w" podStartSLOduration=2.246601502 podStartE2EDuration="2.246601502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:35.244194562 +0000 UTC m=+1179.014260923" watchObservedRunningTime="2026-09-11 18:13:35.246601502 +0000 UTC m=+1179.016667850"
Sep 11 18:13:35.890730 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:35.890641    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:35.895685 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:35.895659    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:36.226964 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:36.226932    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:36.227948 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:36.227928    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:13:38.172728 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:38.172695    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:13:38.173096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:38.172887    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:13:52.054952 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.054917    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"]
Sep 11 18:13:52.058727 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.058710    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.062237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.062220    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-90057a-openshift-default-dockercfg-4dwgp\""
Sep 11 18:13:52.077278 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.077258    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"]
Sep 11 18:13:52.095631 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-token\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.095726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095643    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-certs\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.095726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-podinfo\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.095821 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095732    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vjs4\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-kube-api-access-6vjs4\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.095821 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-envoy\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.095821 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-data\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.095928 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095836    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istiod-ca-cert\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.095928 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095864    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-socket\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.095928 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.095898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-credential-socket\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196287 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196252    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istiod-ca-cert\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196484 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196319    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-socket\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196484 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196353    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-credential-socket\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196484 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196377    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-token\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196484 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-certs\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196484 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196460    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-podinfo\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196484 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vjs4\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-kube-api-access-6vjs4\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196512    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-envoy\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196583    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-data\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196783    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-socket\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.196963 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.196951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-data\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.197077 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.197052    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-certs\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.197152 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.197072    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istiod-ca-cert\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.197230 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.197213    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-credential-socket\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.199350 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.199328    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-envoy\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.199677 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.199658    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-podinfo\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.205818 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.205794    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-token\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.206689 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.206671    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vjs4\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-kube-api-access-6vjs4\") pod \"e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") " pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.369846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.369773    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:52.650214 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.650172    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"]
Sep 11 18:13:52.650685 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:13:52.650647    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c238199_9a97_4c3d_a5d2_a0d7d367c7ef.slice/crio-ffff6d7209a27e5d7fbc3afe2294e11b9667169be38f1581f143773268c7b5b5 WatchSource:0}: Error finding container ffff6d7209a27e5d7fbc3afe2294e11b9667169be38f1581f143773268c7b5b5: Status 404 returned error can't find the container with id ffff6d7209a27e5d7fbc3afe2294e11b9667169be38f1581f143773268c7b5b5
Sep 11 18:13:52.653748 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.653709    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:13:52.653858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.653795    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:13:52.653858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:52.653837    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:13:53.340240 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:53.340198    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" event={"ID":"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef","Type":"ContainerStarted","Data":"125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee"}
Sep 11 18:13:53.340640 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:53.340258    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" event={"ID":"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef","Type":"ContainerStarted","Data":"ffff6d7209a27e5d7fbc3afe2294e11b9667169be38f1581f143773268c7b5b5"}
Sep 11 18:13:53.365949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:53.365900    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podStartSLOduration=1.3658853 podStartE2EDuration="1.3658853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:53.364846437 +0000 UTC m=+1197.134912780" watchObservedRunningTime="2026-09-11 18:13:53.3658853 +0000 UTC m=+1197.135951648"
Sep 11 18:13:53.370196 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:53.370173    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:53.375186 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:53.375164    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:54.346673 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:54.346642    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:13:54.347560 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:13:54.347533    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:14:25.388479 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:25.388436    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 18:14:25.389004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:25.388516    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 18:14:26.217268 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.217225    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-95d118"]
Sep 11 18:14:26.220965 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.220938    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-95d118"
Sep 11 18:14:26.223975 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.223957    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q6zqg\""
Sep 11 18:14:26.231597 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.231566    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-95d118"]
Sep 11 18:14:26.337004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.336966    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzr2j\" (UniqueName: \"kubernetes.io/projected/67e150a1-ec87-4b23-85cf-e7a34d1972ad-kube-api-access-fzr2j\") pod \"test-curl-7496-95d118\" (UID: \"67e150a1-ec87-4b23-85cf-e7a34d1972ad\") " pod="opendatahub/test-curl-7496-95d118"
Sep 11 18:14:26.438405 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.438364    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzr2j\" (UniqueName: \"kubernetes.io/projected/67e150a1-ec87-4b23-85cf-e7a34d1972ad-kube-api-access-fzr2j\") pod \"test-curl-7496-95d118\" (UID: \"67e150a1-ec87-4b23-85cf-e7a34d1972ad\") " pod="opendatahub/test-curl-7496-95d118"
Sep 11 18:14:26.450351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.450315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzr2j\" (UniqueName: \"kubernetes.io/projected/67e150a1-ec87-4b23-85cf-e7a34d1972ad-kube-api-access-fzr2j\") pod \"test-curl-7496-95d118\" (UID: \"67e150a1-ec87-4b23-85cf-e7a34d1972ad\") " pod="opendatahub/test-curl-7496-95d118"
Sep 11 18:14:26.531424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.531394    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-95d118"
Sep 11 18:14:26.562638 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.562525    2571 generic.go:356] "Generic (PLEG): container finished" podID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerID="125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee" exitCode=0
Sep 11 18:14:26.562638 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.562584    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" event={"ID":"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef","Type":"ContainerDied","Data":"125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee"}
Sep 11 18:14:26.562979 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.562957    2571 scope.go:117] "RemoveContainer" containerID="125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee"
Sep 11 18:14:26.564079 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.564043    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:14:26.564195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.564140    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:14:26.564195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.564186    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:14:26.805663 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:26.805631    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-95d118"]
Sep 11 18:14:26.807394 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:14:26.807368    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e150a1_ec87_4b23_85cf_e7a34d1972ad.slice/crio-509e4df8d4cd7c7a89b65f7fd1ef519cd7c6d860e24e1a3bc1f965958115bb6c WatchSource:0}: Error finding container 509e4df8d4cd7c7a89b65f7fd1ef519cd7c6d860e24e1a3bc1f965958115bb6c: Status 404 returned error can't find the container with id 509e4df8d4cd7c7a89b65f7fd1ef519cd7c6d860e24e1a3bc1f965958115bb6c
Sep 11 18:14:27.575207 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:27.575164    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-95d118" event={"ID":"67e150a1-ec87-4b23-85cf-e7a34d1972ad","Type":"ContainerStarted","Data":"59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192"}
Sep 11 18:14:27.575856 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:27.575823    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-95d118" event={"ID":"67e150a1-ec87-4b23-85cf-e7a34d1972ad","Type":"ContainerStarted","Data":"509e4df8d4cd7c7a89b65f7fd1ef519cd7c6d860e24e1a3bc1f965958115bb6c"}
Sep 11 18:14:27.578518 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:27.578496    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" event={"ID":"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef","Type":"ContainerStarted","Data":"3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d"}
Sep 11 18:14:27.592000 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:27.591959    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7496-95d118" podStartSLOduration=1.5919471029999999 podStartE2EDuration="1.591947103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:27.590708995 +0000 UTC m=+1231.360775357" watchObservedRunningTime="2026-09-11 18:14:27.591947103 +0000 UTC m=+1231.362013451"
Sep 11 18:14:28.206885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.206847    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-95d118"]
Sep 11 18:14:28.215147 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.215118    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-95d118"]
Sep 11 18:14:28.335170 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.335089    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-c217b8"]
Sep 11 18:14:28.335599 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.335583    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e150a1-ec87-4b23-85cf-e7a34d1972ad" containerName="test-curl-7496-95d118"
Sep 11 18:14:28.335655 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.335601    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e150a1-ec87-4b23-85cf-e7a34d1972ad" containerName="test-curl-7496-95d118"
Sep 11 18:14:28.335692 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.335669    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e150a1-ec87-4b23-85cf-e7a34d1972ad" containerName="test-curl-7496-95d118"
Sep 11 18:14:28.338847 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.338831    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-c217b8"
Sep 11 18:14:28.351322 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.348114    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-c217b8"]
Sep 11 18:14:28.372323 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.371486    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:14:28.372323 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.371837    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 11 18:14:28.372323 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.371885    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 11 18:14:28.477865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.477823    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6w8t\" (UniqueName: \"kubernetes.io/projected/f5300fc0-4a4b-4d9c-bb93-e3ec878e0333-kube-api-access-q6w8t\") pod \"test-curl-7496-c217b8\" (UID: \"f5300fc0-4a4b-4d9c-bb93-e3ec878e0333\") " pod="opendatahub/test-curl-7496-c217b8"
Sep 11 18:14:28.578726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.578689    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6w8t\" (UniqueName: \"kubernetes.io/projected/f5300fc0-4a4b-4d9c-bb93-e3ec878e0333-kube-api-access-q6w8t\") pod \"test-curl-7496-c217b8\" (UID: \"f5300fc0-4a4b-4d9c-bb93-e3ec878e0333\") " pod="opendatahub/test-curl-7496-c217b8"
Sep 11 18:14:28.584615 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:28.584595    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7496-95d118" container="test-curl-7496-95d118"
Sep 11 18:14:28.589726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.589666    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6w8t\" (UniqueName: \"kubernetes.io/projected/f5300fc0-4a4b-4d9c-bb93-e3ec878e0333-kube-api-access-q6w8t\") pod \"test-curl-7496-c217b8\" (UID: \"f5300fc0-4a4b-4d9c-bb93-e3ec878e0333\") " pod="opendatahub/test-curl-7496-c217b8"
Sep 11 18:14:28.649865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.649829    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-c217b8"
Sep 11 18:14:28.946645 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:28.946623    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-c217b8"]
Sep 11 18:14:28.951044 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:14:28.951006    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5300fc0_4a4b_4d9c_bb93_e3ec878e0333.slice/crio-2c7753543ff47a722cca4817e8cb38b2019c4a7c2e0ab2a4b0c6489053cab655 WatchSource:0}: Error finding container 2c7753543ff47a722cca4817e8cb38b2019c4a7c2e0ab2a4b0c6489053cab655: Status 404 returned error can't find the container with id 2c7753543ff47a722cca4817e8cb38b2019c4a7c2e0ab2a4b0c6489053cab655
Sep 11 18:14:29.371230 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:29.371124    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 11 18:14:29.371230 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:29.371201    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 11 18:14:29.592422 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:29.592388    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-c217b8" event={"ID":"f5300fc0-4a4b-4d9c-bb93-e3ec878e0333","Type":"ContainerStarted","Data":"e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9"}
Sep 11 18:14:29.592422 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:29.592430    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-c217b8" event={"ID":"f5300fc0-4a4b-4d9c-bb93-e3ec878e0333","Type":"ContainerStarted","Data":"2c7753543ff47a722cca4817e8cb38b2019c4a7c2e0ab2a4b0c6489053cab655"}
Sep 11 18:14:29.592981 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:29.592462    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7496-95d118" podUID="67e150a1-ec87-4b23-85cf-e7a34d1972ad" containerName="test-curl-7496-95d118" containerID="cri-o://59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192" gracePeriod=2
Sep 11 18:14:29.610376 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:29.610319    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7496-c217b8" podStartSLOduration=1.610282202 podStartE2EDuration="1.610282202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:29.608580883 +0000 UTC m=+1233.378647233" watchObservedRunningTime="2026-09-11 18:14:29.610282202 +0000 UTC m=+1233.380348549"
Sep 11 18:14:30.267662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.267616    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-c217b8"]
Sep 11 18:14:30.274754 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.274719    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-c217b8"]
Sep 11 18:14:30.373791 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.373695    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 11 18:14:30.374079 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.374049    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 11 18:14:30.388702 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.388677    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-07f72c"]
Sep 11 18:14:30.389344 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.389328    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5300fc0-4a4b-4d9c-bb93-e3ec878e0333" containerName="test-curl-7496-c217b8"
Sep 11 18:14:30.389454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.389446    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5300fc0-4a4b-4d9c-bb93-e3ec878e0333" containerName="test-curl-7496-c217b8"
Sep 11 18:14:30.389595 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.389587    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5300fc0-4a4b-4d9c-bb93-e3ec878e0333" containerName="test-curl-7496-c217b8"
Sep 11 18:14:30.392764 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.392749    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:30.403214 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.403191    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-07f72c"]
Sep 11 18:14:30.495010 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.494966    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm7cc\" (UniqueName: \"kubernetes.io/projected/3efcd133-3449-40d4-948a-b3467409e315-kube-api-access-bm7cc\") pod \"test-curl-7496-07f72c\" (UID: \"3efcd133-3449-40d4-948a-b3467409e315\") " pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:30.596291 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.596252    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7cc\" (UniqueName: \"kubernetes.io/projected/3efcd133-3449-40d4-948a-b3467409e315-kube-api-access-bm7cc\") pod \"test-curl-7496-07f72c\" (UID: \"3efcd133-3449-40d4-948a-b3467409e315\") " pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:30.599320 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:30.599284    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7496-c217b8" container="test-curl-7496-c217b8"
Sep 11 18:14:30.606543 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.606518    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7cc\" (UniqueName: \"kubernetes.io/projected/3efcd133-3449-40d4-948a-b3467409e315-kube-api-access-bm7cc\") pod \"test-curl-7496-07f72c\" (UID: \"3efcd133-3449-40d4-948a-b3467409e315\") " pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:30.702510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:30.702484    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:31.021634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.021552    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-07f72c"]
Sep 11 18:14:31.025315 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:14:31.025277    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3efcd133_3449_40d4_948a_b3467409e315.slice/crio-89712725c2583f9123e7f5ef453586162029d9b1a8ab8dd523084b891e4645db WatchSource:0}: Error finding container 89712725c2583f9123e7f5ef453586162029d9b1a8ab8dd523084b891e4645db: Status 404 returned error can't find the container with id 89712725c2583f9123e7f5ef453586162029d9b1a8ab8dd523084b891e4645db
Sep 11 18:14:31.370805 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.370713    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 11 18:14:31.371061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.371029    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 11 18:14:31.607593 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.607564    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-07f72c" event={"ID":"3efcd133-3449-40d4-948a-b3467409e315","Type":"ContainerStarted","Data":"1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd"}
Sep 11 18:14:31.607874 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.607592    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7496-c217b8" podUID="f5300fc0-4a4b-4d9c-bb93-e3ec878e0333" containerName="test-curl-7496-c217b8" containerID="cri-o://e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9" gracePeriod=2
Sep 11 18:14:31.607874 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.607608    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-07f72c" event={"ID":"3efcd133-3449-40d4-948a-b3467409e315","Type":"ContainerStarted","Data":"89712725c2583f9123e7f5ef453586162029d9b1a8ab8dd523084b891e4645db"}
Sep 11 18:14:31.624525 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.624436    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7496-07f72c" podStartSLOduration=1.624418841 podStartE2EDuration="1.624418841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:31.623060836 +0000 UTC m=+1235.393127185" watchObservedRunningTime="2026-09-11 18:14:31.624418841 +0000 UTC m=+1235.394485190"
Sep 11 18:14:31.787693 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.787665    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-95d118"
Sep 11 18:14:31.807776 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.807755    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzr2j\" (UniqueName: \"kubernetes.io/projected/67e150a1-ec87-4b23-85cf-e7a34d1972ad-kube-api-access-fzr2j\") pod \"67e150a1-ec87-4b23-85cf-e7a34d1972ad\" (UID: \"67e150a1-ec87-4b23-85cf-e7a34d1972ad\") "
Sep 11 18:14:31.809793 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.809766    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e150a1-ec87-4b23-85cf-e7a34d1972ad-kube-api-access-fzr2j" (OuterVolumeSpecName: "kube-api-access-fzr2j") pod "67e150a1-ec87-4b23-85cf-e7a34d1972ad" (UID: "67e150a1-ec87-4b23-85cf-e7a34d1972ad"). InnerVolumeSpecName "kube-api-access-fzr2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:31.909075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:31.909046    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzr2j\" (UniqueName: \"kubernetes.io/projected/67e150a1-ec87-4b23-85cf-e7a34d1972ad-kube-api-access-fzr2j\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:14:32.253440 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.253288    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-07f72c"]
Sep 11 18:14:32.261360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.261326    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-07f72c"]
Sep 11 18:14:32.374358 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.370878    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Sep 11 18:14:32.374358 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.370942    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Sep 11 18:14:32.379465 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.379439    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7496-3af6e1"]
Sep 11 18:14:32.379907 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.379894    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3efcd133-3449-40d4-948a-b3467409e315" containerName="test-curl-7496-07f72c"
Sep 11 18:14:32.379954 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.379909    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3efcd133-3449-40d4-948a-b3467409e315" containerName="test-curl-7496-07f72c"
Sep 11 18:14:32.379993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.379982    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3efcd133-3449-40d4-948a-b3467409e315" containerName="test-curl-7496-07f72c"
Sep 11 18:14:32.383327 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.383234    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-3af6e1"
Sep 11 18:14:32.385857 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.385833    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-3af6e1"]
Sep 11 18:14:32.413778 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.413743    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbqh\" (UniqueName: \"kubernetes.io/projected/1c262518-9925-4c98-b7d1-9ebb589b5b56-kube-api-access-nmbqh\") pod \"test-curl-7496-3af6e1\" (UID: \"1c262518-9925-4c98-b7d1-9ebb589b5b56\") " pod="opendatahub/test-curl-7496-3af6e1"
Sep 11 18:14:32.514882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.514797    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbqh\" (UniqueName: \"kubernetes.io/projected/1c262518-9925-4c98-b7d1-9ebb589b5b56-kube-api-access-nmbqh\") pod \"test-curl-7496-3af6e1\" (UID: \"1c262518-9925-4c98-b7d1-9ebb589b5b56\") " pod="opendatahub/test-curl-7496-3af6e1"
Sep 11 18:14:32.524251 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.524226    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbqh\" (UniqueName: \"kubernetes.io/projected/1c262518-9925-4c98-b7d1-9ebb589b5b56-kube-api-access-nmbqh\") pod \"test-curl-7496-3af6e1\" (UID: \"1c262518-9925-4c98-b7d1-9ebb589b5b56\") " pod="opendatahub/test-curl-7496-3af6e1"
Sep 11 18:14:32.615450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.615412    2571 generic.go:356] "Generic (PLEG): container finished" podID="67e150a1-ec87-4b23-85cf-e7a34d1972ad" containerID="59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192" exitCode=137
Sep 11 18:14:32.615880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.615466    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-95d118"
Sep 11 18:14:32.615880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.615513    2571 scope.go:117] "RemoveContainer" containerID="59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192"
Sep 11 18:14:32.615880 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:32.615787    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7496-07f72c" container="test-curl-7496-07f72c"
Sep 11 18:14:32.627551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.627528    2571 scope.go:117] "RemoveContainer" containerID="59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192"
Sep 11 18:14:32.627632 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:32.627606    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-95d118\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="67e150a1-ec87-4b23-85cf-e7a34d1972ad" pod="opendatahub/test-curl-7496-95d118"
Sep 11 18:14:32.627824 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:32.627806    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192\": container with ID starting with 59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192 not found: ID does not exist" containerID="59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192"
Sep 11 18:14:32.627868 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.627834    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192"} err="failed to get container status \"59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192\": rpc error: code = NotFound desc = could not find container \"59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192\": container with ID starting with 59cacc2b029188ee30c41a1156ccafb23eb66be92b9555135d7b90b40625a192 not found: ID does not exist"
Sep 11 18:14:32.693408 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.693372    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-3af6e1"
Sep 11 18:14:32.733673 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.733645    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e150a1-ec87-4b23-85cf-e7a34d1972ad" path="/var/lib/kubelet/pods/67e150a1-ec87-4b23-85cf-e7a34d1972ad/volumes"
Sep 11 18:14:32.975990 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:32.975963    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7496-3af6e1"]
Sep 11 18:14:32.976975 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:14:32.976943    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c262518_9925_4c98_b7d1_9ebb589b5b56.slice/crio-1e325da11a17645c8ade9700347062a10e009d15a621da0112335369841b85d9 WatchSource:0}: Error finding container 1e325da11a17645c8ade9700347062a10e009d15a621da0112335369841b85d9: Status 404 returned error can't find the container with id 1e325da11a17645c8ade9700347062a10e009d15a621da0112335369841b85d9
Sep 11 18:14:33.373908 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.373789    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:14:33.374355 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.374336    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:14:33.374952 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.374936    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:14:33.624763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.624697    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-3af6e1" event={"ID":"1c262518-9925-4c98-b7d1-9ebb589b5b56","Type":"ContainerStarted","Data":"df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e"}
Sep 11 18:14:33.624763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.624746    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7496-3af6e1" event={"ID":"1c262518-9925-4c98-b7d1-9ebb589b5b56","Type":"ContainerStarted","Data":"1e325da11a17645c8ade9700347062a10e009d15a621da0112335369841b85d9"}
Sep 11 18:14:33.625117 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.624755    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7496-07f72c" podUID="3efcd133-3449-40d4-948a-b3467409e315" containerName="test-curl-7496-07f72c" containerID="cri-o://1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd" gracePeriod=2
Sep 11 18:14:33.642967 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.642924    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7496-3af6e1" podStartSLOduration=1.6429054939999999 podStartE2EDuration="1.642905494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:33.642343906 +0000 UTC m=+1237.412410254" watchObservedRunningTime="2026-09-11 18:14:33.642905494 +0000 UTC m=+1237.412971842"
Sep 11 18:14:33.796873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.796849    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-c217b8"
Sep 11 18:14:33.826642 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.826609    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6w8t\" (UniqueName: \"kubernetes.io/projected/f5300fc0-4a4b-4d9c-bb93-e3ec878e0333-kube-api-access-q6w8t\") pod \"f5300fc0-4a4b-4d9c-bb93-e3ec878e0333\" (UID: \"f5300fc0-4a4b-4d9c-bb93-e3ec878e0333\") "
Sep 11 18:14:33.828736 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.828712    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5300fc0-4a4b-4d9c-bb93-e3ec878e0333-kube-api-access-q6w8t" (OuterVolumeSpecName: "kube-api-access-q6w8t") pod "f5300fc0-4a4b-4d9c-bb93-e3ec878e0333" (UID: "f5300fc0-4a4b-4d9c-bb93-e3ec878e0333"). InnerVolumeSpecName "kube-api-access-q6w8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:33.927885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:33.927854    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6w8t\" (UniqueName: \"kubernetes.io/projected/f5300fc0-4a4b-4d9c-bb93-e3ec878e0333-kube-api-access-q6w8t\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:14:34.287220 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:34.287124    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7496-3af6e1"]
Sep 11 18:14:34.299944 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:34.299908    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7496-3af6e1"]
Sep 11 18:14:34.632626 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:34.632537    2571 generic.go:356] "Generic (PLEG): container finished" podID="f5300fc0-4a4b-4d9c-bb93-e3ec878e0333" containerID="e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9" exitCode=137
Sep 11 18:14:34.632626 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:34.632588    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-c217b8"
Sep 11 18:14:34.632626 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:34.632610    2571 scope.go:117] "RemoveContainer" containerID="e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9"
Sep 11 18:14:34.633158 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:34.632972    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7496-3af6e1" container="test-curl-7496-3af6e1"
Sep 11 18:14:34.642274 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:34.642247    2571 scope.go:117] "RemoveContainer" containerID="e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9"
Sep 11 18:14:34.642562 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:34.642544    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9\": container with ID starting with e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9 not found: ID does not exist" containerID="e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9"
Sep 11 18:14:34.642617 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:34.642570    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9"} err="failed to get container status \"e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9\": rpc error: code = NotFound desc = could not find container \"e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9\": container with ID starting with e3fcf9fc5e7bb21fbc6cdce8fc631335d7806445d05d7c85e4e177944b586be9 not found: ID does not exist"
Sep 11 18:14:34.643087 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:34.643063    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-c217b8\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="f5300fc0-4a4b-4d9c-bb93-e3ec878e0333" pod="opendatahub/test-curl-7496-c217b8"
Sep 11 18:14:34.731251 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:34.731216    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5300fc0-4a4b-4d9c-bb93-e3ec878e0333" path="/var/lib/kubelet/pods/f5300fc0-4a4b-4d9c-bb93-e3ec878e0333/volumes"
Sep 11 18:14:35.193156 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.193118    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"]
Sep 11 18:14:35.193628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.193611    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c262518-9925-4c98-b7d1-9ebb589b5b56" containerName="test-curl-7496-3af6e1"
Sep 11 18:14:35.193678 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.193630    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c262518-9925-4c98-b7d1-9ebb589b5b56" containerName="test-curl-7496-3af6e1"
Sep 11 18:14:35.193725 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.193715    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c262518-9925-4c98-b7d1-9ebb589b5b56" containerName="test-curl-7496-3af6e1"
Sep 11 18:14:35.198352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.198336    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.201515 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.201497    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3a652320-openshift-default-dockercfg-k99bh\""
Sep 11 18:14:35.210285 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.210258    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"]
Sep 11 18:14:35.238782 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.238748    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-certs\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.238973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.238803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-token\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.238973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.238860    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istiod-ca-cert\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.238973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.238898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r74w\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-kube-api-access-4r74w\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.238973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.238921    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-credential-socket\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.238973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.238956    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-data\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.239271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.239006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-socket\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.239271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.239029    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-podinfo\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.239271 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.239088    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-envoy\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340233 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340199    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-certs\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340249    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-token\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340270    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istiod-ca-cert\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340287    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r74w\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-kube-api-access-4r74w\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340316    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-credential-socket\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340350    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-data\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340515    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-socket\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340584    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-podinfo\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340650    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-envoy\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340687    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-credential-socket\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340735    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-certs\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.340929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340770    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-data\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.341199 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.340928    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-socket\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.341199 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.341161    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istiod-ca-cert\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.342853 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.342831    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-envoy\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.343049 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.343034    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-podinfo\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.348974 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.348953    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-token\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.349418 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.349396    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r74w\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-kube-api-access-4r74w\") pod \"e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") " pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.508827 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.508740    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:35.640592 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.640561    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7496-3af6e1" podUID="1c262518-9925-4c98-b7d1-9ebb589b5b56" containerName="test-curl-7496-3af6e1" containerID="cri-o://df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e" gracePeriod=2
Sep 11 18:14:35.791863 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.791835    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:35.794625 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.794602    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"]
Sep 11 18:14:35.796087 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:14:35.796063    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b7a058_ba23_48ab_a0ce_8fbfbc7db0fd.slice/crio-3ea7faa00d624cd09c4e3e2e4d23492199cec3f98384066eebf28a51290bcbb5 WatchSource:0}: Error finding container 3ea7faa00d624cd09c4e3e2e4d23492199cec3f98384066eebf28a51290bcbb5: Status 404 returned error can't find the container with id 3ea7faa00d624cd09c4e3e2e4d23492199cec3f98384066eebf28a51290bcbb5
Sep 11 18:14:35.798485 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.798450    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:14:35.798583 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.798512    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:14:35.798583 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.798552    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:14:35.845444 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.845421    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm7cc\" (UniqueName: \"kubernetes.io/projected/3efcd133-3449-40d4-948a-b3467409e315-kube-api-access-bm7cc\") pod \"3efcd133-3449-40d4-948a-b3467409e315\" (UID: \"3efcd133-3449-40d4-948a-b3467409e315\") "
Sep 11 18:14:35.847771 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.847732    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efcd133-3449-40d4-948a-b3467409e315-kube-api-access-bm7cc" (OuterVolumeSpecName: "kube-api-access-bm7cc") pod "3efcd133-3449-40d4-948a-b3467409e315" (UID: "3efcd133-3449-40d4-948a-b3467409e315"). InnerVolumeSpecName "kube-api-access-bm7cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:35.947258 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:35.947216    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm7cc\" (UniqueName: \"kubernetes.io/projected/3efcd133-3449-40d4-948a-b3467409e315-kube-api-access-bm7cc\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:14:36.649099 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.649001    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" event={"ID":"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd","Type":"ContainerStarted","Data":"2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8"}
Sep 11 18:14:36.649099 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.649067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" event={"ID":"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd","Type":"ContainerStarted","Data":"3ea7faa00d624cd09c4e3e2e4d23492199cec3f98384066eebf28a51290bcbb5"}
Sep 11 18:14:36.650592 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.650563    2571 generic.go:356] "Generic (PLEG): container finished" podID="3efcd133-3449-40d4-948a-b3467409e315" containerID="1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd" exitCode=137
Sep 11 18:14:36.650746 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.650617    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:36.650746 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.650660    2571 scope.go:117] "RemoveContainer" containerID="1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd"
Sep 11 18:14:36.661264 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.661247    2571 scope.go:117] "RemoveContainer" containerID="1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd"
Sep 11 18:14:36.661564 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:36.661542    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd\": container with ID starting with 1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd not found: ID does not exist" containerID="1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd"
Sep 11 18:14:36.661645 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.661571    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd"} err="failed to get container status \"1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd\": rpc error: code = NotFound desc = could not find container \"1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd\": container with ID starting with 1ab1d6b0f836de4b21c44a38746a9c1da465bd26223c16029b9a1ca15c7673dd not found: ID does not exist"
Sep 11 18:14:36.675475 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:36.675444    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-07f72c\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="3efcd133-3449-40d4-948a-b3467409e315" pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:36.675612 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.675573    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" podStartSLOduration=1.6755580669999999 podStartE2EDuration="1.675558067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:36.673011186 +0000 UTC m=+1240.443077547" watchObservedRunningTime="2026-09-11 18:14:36.675558067 +0000 UTC m=+1240.445624414"
Sep 11 18:14:36.730606 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:36.730564    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-07f72c\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="3efcd133-3449-40d4-948a-b3467409e315" pod="opendatahub/test-curl-7496-07f72c"
Sep 11 18:14:36.732472 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:36.732443    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3efcd133-3449-40d4-948a-b3467409e315" path="/var/lib/kubelet/pods/3efcd133-3449-40d4-948a-b3467409e315/volumes"
Sep 11 18:14:37.509843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:37.509809    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:37.515767 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:37.515739    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:37.658950 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:37.658927    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:37.660176 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:37.660155    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:14:37.843232 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:37.843208    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-3af6e1"
Sep 11 18:14:37.867618 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:37.867587    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmbqh\" (UniqueName: \"kubernetes.io/projected/1c262518-9925-4c98-b7d1-9ebb589b5b56-kube-api-access-nmbqh\") pod \"1c262518-9925-4c98-b7d1-9ebb589b5b56\" (UID: \"1c262518-9925-4c98-b7d1-9ebb589b5b56\") "
Sep 11 18:14:37.869716 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:37.869693    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c262518-9925-4c98-b7d1-9ebb589b5b56-kube-api-access-nmbqh" (OuterVolumeSpecName: "kube-api-access-nmbqh") pod "1c262518-9925-4c98-b7d1-9ebb589b5b56" (UID: "1c262518-9925-4c98-b7d1-9ebb589b5b56"). InnerVolumeSpecName "kube-api-access-nmbqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:14:37.968835 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:37.968799    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbqh\" (UniqueName: \"kubernetes.io/projected/1c262518-9925-4c98-b7d1-9ebb589b5b56-kube-api-access-nmbqh\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:14:38.666462 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:38.666430    2571 generic.go:356] "Generic (PLEG): container finished" podID="1c262518-9925-4c98-b7d1-9ebb589b5b56" containerID="df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e" exitCode=137
Sep 11 18:14:38.666915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:38.666475    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7496-3af6e1"
Sep 11 18:14:38.666915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:38.666536    2571 scope.go:117] "RemoveContainer" containerID="df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e"
Sep 11 18:14:38.676021 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:38.676003    2571 scope.go:117] "RemoveContainer" containerID="df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e"
Sep 11 18:14:38.676307 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:38.676275    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e\": container with ID starting with df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e not found: ID does not exist" containerID="df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e"
Sep 11 18:14:38.676378 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:38.676325    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e"} err="failed to get container status \"df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e\": rpc error: code = NotFound desc = could not find container \"df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e\": container with ID starting with df985133c954ef4abc7561d547dc2c5da260131340963159a79b8dc14b1c790e not found: ID does not exist"
Sep 11 18:14:38.677101 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:14:38.677079    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7496-3af6e1\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="1c262518-9925-4c98-b7d1-9ebb589b5b56" pod="opendatahub/test-curl-7496-3af6e1"
Sep 11 18:14:38.731219 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:38.731187    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c262518-9925-4c98-b7d1-9ebb589b5b56" path="/var/lib/kubelet/pods/1c262518-9925-4c98-b7d1-9ebb589b5b56/volumes"
Sep 11 18:14:42.514718 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.514680    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"]
Sep 11 18:14:42.519867 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.519848    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.523157 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.523134    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cd668b\"/\"kube-root-ca.crt\""
Sep 11 18:14:42.524335 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.524315    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cd668b\"/\"noten-model-cd668b-kserve-self-signed-certs\""
Sep 11 18:14:42.524489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.524468    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-cd668b\"/\"openshift-service-ca.crt\""
Sep 11 18:14:42.524535 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.524471    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-cd668b\"/\"default-dockercfg-vsx52\""
Sep 11 18:14:42.528939 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.528907    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"]
Sep 11 18:14:42.615756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.615720    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-home\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.615756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.615760    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-model-cache\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.615951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.615792    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a04df8d-d218-47b4-a898-6ccb92c32246-tls-certs\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.615951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.615887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-dshm\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.616038 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.615951    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-tmp-dir\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.616038 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.615991    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssxp7\" (UniqueName: \"kubernetes.io/projected/5a04df8d-d218-47b4-a898-6ccb92c32246-kube-api-access-ssxp7\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.717388 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.717346    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a04df8d-d218-47b4-a898-6ccb92c32246-tls-certs\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.717577 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.717438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-dshm\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.717577 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.717481    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-tmp-dir\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.717577 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.717519    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssxp7\" (UniqueName: \"kubernetes.io/projected/5a04df8d-d218-47b4-a898-6ccb92c32246-kube-api-access-ssxp7\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.717577 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.717547    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-home\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.717577 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.717561    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-model-cache\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.717992 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.717971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-tmp-dir\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.717992 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.717984    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-home\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.718128 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.718045    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-model-cache\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.719693 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.719670    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-dshm\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.720016 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.719997    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a04df8d-d218-47b4-a898-6ccb92c32246-tls-certs\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.727179 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.727152    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssxp7\" (UniqueName: \"kubernetes.io/projected/5a04df8d-d218-47b4-a898-6ccb92c32246-kube-api-access-ssxp7\") pod \"noten-model-cd668b-kserve-8c5c8647f-r88c8\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") " pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:42.831393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:42.831283    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:43.111023 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:43.110943    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"]
Sep 11 18:14:43.115642 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:14:43.115606    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a04df8d_d218_47b4_a898_6ccb92c32246.slice/crio-21844b91862754e4fa67bc912ef25940591093662a7d58de84293d168bab0c9e WatchSource:0}: Error finding container 21844b91862754e4fa67bc912ef25940591093662a7d58de84293d168bab0c9e: Status 404 returned error can't find the container with id 21844b91862754e4fa67bc912ef25940591093662a7d58de84293d168bab0c9e
Sep 11 18:14:43.707351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:43.707292    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8" event={"ID":"5a04df8d-d218-47b4-a898-6ccb92c32246","Type":"ContainerStarted","Data":"9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e"}
Sep 11 18:14:43.707856 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:43.707364    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8" event={"ID":"5a04df8d-d218-47b4-a898-6ccb92c32246","Type":"ContainerStarted","Data":"21844b91862754e4fa67bc912ef25940591093662a7d58de84293d168bab0c9e"}
Sep 11 18:14:45.301083 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.301026    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8" podStartSLOduration=3.30100615 podStartE2EDuration="3.30100615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:43.748580051 +0000 UTC m=+1247.518646400" watchObservedRunningTime="2026-09-11 18:14:45.30100615 +0000 UTC m=+1249.071072494"
Sep 11 18:14:45.301509 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.301192    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"]
Sep 11 18:14:45.305021 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.305005    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:45.308483 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.308462    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3a652320\""
Sep 11 18:14:45.308985 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.308972    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3a652320-dockercfg-pjsdq\""
Sep 11 18:14:45.315111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.315089    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"]
Sep 11 18:14:45.454787 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.454744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbs9\" (UniqueName: \"kubernetes.io/projected/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-kube-api-access-mgbs9\") pod \"payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8\" (UID: \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:45.454957 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.454806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8\" (UID: \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:45.556365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.556268    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbs9\" (UniqueName: \"kubernetes.io/projected/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-kube-api-access-mgbs9\") pod \"payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8\" (UID: \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:45.556365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.556349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8\" (UID: \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:45.557080 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.557060    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8\" (UID: \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:45.566343 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.566316    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbs9\" (UniqueName: \"kubernetes.io/projected/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-kube-api-access-mgbs9\") pod \"payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8\" (UID: \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:45.617459 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.617420    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:45.897109 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:45.897078    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"]
Sep 11 18:14:45.897427 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:14:45.897407    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73ca6b4_2f05_4e41_be7f_c1bd050d65bc.slice/crio-98cb19518df506a9ffc68a435d5dcab63c25fc9dea19c00e5fd514c8126f820c WatchSource:0}: Error finding container 98cb19518df506a9ffc68a435d5dcab63c25fc9dea19c00e5fd514c8126f820c: Status 404 returned error can't find the container with id 98cb19518df506a9ffc68a435d5dcab63c25fc9dea19c00e5fd514c8126f820c
Sep 11 18:14:46.736845 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:46.736655    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:14:46.736845 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:46.736723    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8" event={"ID":"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc","Type":"ContainerStarted","Data":"191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23"}
Sep 11 18:14:46.736845 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:46.736806    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8" event={"ID":"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc","Type":"ContainerStarted","Data":"98cb19518df506a9ffc68a435d5dcab63c25fc9dea19c00e5fd514c8126f820c"}
Sep 11 18:14:46.756141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:46.756090    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8" podStartSLOduration=1.756076178 podStartE2EDuration="1.756076178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:46.753133018 +0000 UTC m=+1250.523199378" watchObservedRunningTime="2026-09-11 18:14:46.756076178 +0000 UTC m=+1250.526142527"
Sep 11 18:14:52.832850 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:52.832757    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:52.832850 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:52.832814    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:52.845397 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:52.845373    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:53.791426 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:53.791393    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:14:56.935425 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:56.935394    2571 scope.go:117] "RemoveContainer" containerID="b060777644a1796c3f90bd734eda6145889dfb5b129797fd3f9fe6f3f3bfef28"
Sep 11 18:14:56.944238 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:56.944220    2571 scope.go:117] "RemoveContainer" containerID="e47b77997586eb4a2a4c3585dc2b23283d7eb07a26b6c58795c65abf20f4a0b7"
Sep 11 18:14:58.747935 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:14:58.747903    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:15:00.163666 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.163636    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"]
Sep 11 18:15:00.169047 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.169024    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl"]
Sep 11 18:15:00.169181 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.169133    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"
Sep 11 18:15:00.172407 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.172390    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"]
Sep 11 18:15:00.172529 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.172507    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl"
Sep 11 18:15:00.173075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.173054    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:15:00.175781 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.175762    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"
Sep 11 18:15:00.180609 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.180593    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"
Sep 11 18:15:00.188598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.188577    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl"
Sep 11 18:15:00.191679 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.191655    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"]
Sep 11 18:15:00.198700 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.198677    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl"]
Sep 11 18:15:00.199075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.199057    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"
Sep 11 18:15:00.199884 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.199859    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"]
Sep 11 18:15:00.204121 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.204069    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"]
Sep 11 18:15:00.208826 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.208809    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"]
Sep 11 18:15:00.218320 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.217633    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"
Sep 11 18:15:00.223172 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.223150    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"]
Sep 11 18:15:00.226932 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.226905    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"]
Sep 11 18:15:00.232766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.232741    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"
Sep 11 18:15:00.244741 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.244720    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"
Sep 11 18:15:00.299218 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.298994    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"
Sep 11 18:15:00.484751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.484722    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"]
Sep 11 18:15:00.485866 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:15:00.485835    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed94cfb_b010_4aed_bd1b_0e2bfaf8544d.slice/crio-843255bd28a6aefbbf1e7bf1c38effeee577d3c8da7d95ba009b90253f1e14db WatchSource:0}: Error finding container 843255bd28a6aefbbf1e7bf1c38effeee577d3c8da7d95ba009b90253f1e14db: Status 404 returned error can't find the container with id 843255bd28a6aefbbf1e7bf1c38effeee577d3c8da7d95ba009b90253f1e14db
Sep 11 18:15:00.514202 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.514178    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl"]
Sep 11 18:15:00.516430 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:15:00.516405    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c044c58_ecd9_4ba2_a757_029e6975cfae.slice/crio-7acb44cc4b0482ad03c1fe735f4e5757c936bafe6c314cc9f7f25d8024ea5db7 WatchSource:0}: Error finding container 7acb44cc4b0482ad03c1fe735f4e5757c936bafe6c314cc9f7f25d8024ea5db7: Status 404 returned error can't find the container with id 7acb44cc4b0482ad03c1fe735f4e5757c936bafe6c314cc9f7f25d8024ea5db7
Sep 11 18:15:00.559569 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.559544    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"]
Sep 11 18:15:00.560522 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:15:00.560496    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b5f6bf_694f_4238_91bc_c0efc6492347.slice/crio-8f66a3c52868fdb3813b5ce0f78925e6e4f47fd470c67ee82dda835447936598 WatchSource:0}: Error finding container 8f66a3c52868fdb3813b5ce0f78925e6e4f47fd470c67ee82dda835447936598: Status 404 returned error can't find the container with id 8f66a3c52868fdb3813b5ce0f78925e6e4f47fd470c67ee82dda835447936598
Sep 11 18:15:00.573211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.573187    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"]
Sep 11 18:15:00.575641 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:15:00.575615    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43732a78_29e3_40bb_ac1c_b30272c559ca.slice/crio-9c11ee14efb3e108aff3cebec2dc3849b931343df098c510f92f28b400f6159d WatchSource:0}: Error finding container 9c11ee14efb3e108aff3cebec2dc3849b931343df098c510f92f28b400f6159d: Status 404 returned error can't find the container with id 9c11ee14efb3e108aff3cebec2dc3849b931343df098c510f92f28b400f6159d
Sep 11 18:15:00.604891 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.604869    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"]
Sep 11 18:15:00.605345 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:15:00.605323    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6f835f_cb9d_4356_a4bc_7b3c03099802.slice/crio-db6bef29098cf39b2b54ca36eed922130903e61c897760cb12ce48a36998d642 WatchSource:0}: Error finding container db6bef29098cf39b2b54ca36eed922130903e61c897760cb12ce48a36998d642: Status 404 returned error can't find the container with id db6bef29098cf39b2b54ca36eed922130903e61c897760cb12ce48a36998d642
Sep 11 18:15:00.835770 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.835725    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq" event={"ID":"76b5f6bf-694f-4238-91bc-c0efc6492347","Type":"ContainerStarted","Data":"8f66a3c52868fdb3813b5ce0f78925e6e4f47fd470c67ee82dda835447936598"}
Sep 11 18:15:00.836977 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.836952    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl" event={"ID":"4c044c58-ecd9-4ba2-a757-029e6975cfae","Type":"ContainerStarted","Data":"7acb44cc4b0482ad03c1fe735f4e5757c936bafe6c314cc9f7f25d8024ea5db7"}
Sep 11 18:15:00.838456 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.838429    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd" event={"ID":"7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d","Type":"ContainerStarted","Data":"4c4ed22d8d51dd93d41586c64d297948402cbc3913dfab223da75acb92e420e4"}
Sep 11 18:15:00.838582 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.838475    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd" event={"ID":"7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d","Type":"ContainerStarted","Data":"843255bd28a6aefbbf1e7bf1c38effeee577d3c8da7d95ba009b90253f1e14db"}
Sep 11 18:15:00.839701 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.839681    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l" event={"ID":"7b6f835f-cb9d-4356-a4bc-7b3c03099802","Type":"ContainerStarted","Data":"db6bef29098cf39b2b54ca36eed922130903e61c897760cb12ce48a36998d642"}
Sep 11 18:15:00.840800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.840780    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl" event={"ID":"43732a78-29e3-40bb-ac1c-b30272c559ca","Type":"ContainerStarted","Data":"9c11ee14efb3e108aff3cebec2dc3849b931343df098c510f92f28b400f6159d"}
Sep 11 18:15:00.857776 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:00.857726    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd" podStartSLOduration=0.857706011 podStartE2EDuration="857.706011ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:00.857010196 +0000 UTC m=+1264.627076544" watchObservedRunningTime="2026-09-11 18:15:00.857706011 +0000 UTC m=+1264.627772360"
Sep 11 18:15:01.850649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.850604    2571 generic.go:356] "Generic (PLEG): container finished" podID="7b6f835f-cb9d-4356-a4bc-7b3c03099802" containerID="3cd1ad1e4c1682699437f9fe478f118c76fa8852bc8fe3962327dbdbd02fe6c1" exitCode=0
Sep 11 18:15:01.851109 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.850709    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l" event={"ID":"7b6f835f-cb9d-4356-a4bc-7b3c03099802","Type":"ContainerDied","Data":"3cd1ad1e4c1682699437f9fe478f118c76fa8852bc8fe3962327dbdbd02fe6c1"}
Sep 11 18:15:01.853123 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.853092    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl" event={"ID":"43732a78-29e3-40bb-ac1c-b30272c559ca","Type":"ContainerStarted","Data":"933236af869f0b6afbe0ebcf514b725574fcbcd2acfa599ddd967b5dca643240"}
Sep 11 18:15:01.855160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.855135    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq" event={"ID":"76b5f6bf-694f-4238-91bc-c0efc6492347","Type":"ContainerStarted","Data":"65adf3b0dd72511624ad74909eb161fa97748b89de0e2ee04b6474192ec6f4a5"}
Sep 11 18:15:01.857206 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.857172    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl" event={"ID":"4c044c58-ecd9-4ba2-a757-029e6975cfae","Type":"ContainerStarted","Data":"4b51854a60c3f8f06abdd186a85ec6eaef362a82c8e73d14fbea2e1f7eb5f988"}
Sep 11 18:15:01.859224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.859123    2571 generic.go:356] "Generic (PLEG): container finished" podID="7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d" containerID="4c4ed22d8d51dd93d41586c64d297948402cbc3913dfab223da75acb92e420e4" exitCode=0
Sep 11 18:15:01.859224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.859165    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd" event={"ID":"7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d","Type":"ContainerDied","Data":"4c4ed22d8d51dd93d41586c64d297948402cbc3913dfab223da75acb92e420e4"}
Sep 11 18:15:01.911714 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.911666    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq" podStartSLOduration=1.911651232 podStartE2EDuration="1.911651232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:01.911339735 +0000 UTC m=+1265.681406084" watchObservedRunningTime="2026-09-11 18:15:01.911651232 +0000 UTC m=+1265.681717580"
Sep 11 18:15:01.937160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.937098    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl" podStartSLOduration=1.937077599 podStartE2EDuration="1.937077599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:01.93581857 +0000 UTC m=+1265.705885057" watchObservedRunningTime="2026-09-11 18:15:01.937077599 +0000 UTC m=+1265.707143948"
Sep 11 18:15:01.955920 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:01.955865    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl" podStartSLOduration=1.9558477619999999 podStartE2EDuration="1.955847762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:01.952883922 +0000 UTC m=+1265.722950271" watchObservedRunningTime="2026-09-11 18:15:01.955847762 +0000 UTC m=+1265.725914153"
Sep 11 18:15:02.870201 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:02.870169    2571 generic.go:356] "Generic (PLEG): container finished" podID="43732a78-29e3-40bb-ac1c-b30272c559ca" containerID="933236af869f0b6afbe0ebcf514b725574fcbcd2acfa599ddd967b5dca643240" exitCode=0
Sep 11 18:15:02.870646 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:02.870256    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl" event={"ID":"43732a78-29e3-40bb-ac1c-b30272c559ca","Type":"ContainerDied","Data":"933236af869f0b6afbe0ebcf514b725574fcbcd2acfa599ddd967b5dca643240"}
Sep 11 18:15:02.872031 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:02.872008    2571 generic.go:356] "Generic (PLEG): container finished" podID="76b5f6bf-694f-4238-91bc-c0efc6492347" containerID="65adf3b0dd72511624ad74909eb161fa97748b89de0e2ee04b6474192ec6f4a5" exitCode=0
Sep 11 18:15:02.872147 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:02.872096    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq" event={"ID":"76b5f6bf-694f-4238-91bc-c0efc6492347","Type":"ContainerDied","Data":"65adf3b0dd72511624ad74909eb161fa97748b89de0e2ee04b6474192ec6f4a5"}
Sep 11 18:15:02.873695 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:02.873672    2571 generic.go:356] "Generic (PLEG): container finished" podID="4c044c58-ecd9-4ba2-a757-029e6975cfae" containerID="4b51854a60c3f8f06abdd186a85ec6eaef362a82c8e73d14fbea2e1f7eb5f988" exitCode=0
Sep 11 18:15:02.873774 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:02.873757    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl" event={"ID":"4c044c58-ecd9-4ba2-a757-029e6975cfae","Type":"ContainerDied","Data":"4b51854a60c3f8f06abdd186a85ec6eaef362a82c8e73d14fbea2e1f7eb5f988"}
Sep 11 18:15:03.066205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:03.066111    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"
Sep 11 18:15:03.098607 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:03.098581    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"
Sep 11 18:15:03.883863 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:03.883837    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"
Sep 11 18:15:03.884347 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:03.883874    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l" event={"ID":"7b6f835f-cb9d-4356-a4bc-7b3c03099802","Type":"ContainerDied","Data":"db6bef29098cf39b2b54ca36eed922130903e61c897760cb12ce48a36998d642"}
Sep 11 18:15:03.884347 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:03.883923    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db6bef29098cf39b2b54ca36eed922130903e61c897760cb12ce48a36998d642"
Sep 11 18:15:03.885464 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:03.885445    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"
Sep 11 18:15:03.885575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:03.885478    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd" event={"ID":"7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d","Type":"ContainerDied","Data":"843255bd28a6aefbbf1e7bf1c38effeee577d3c8da7d95ba009b90253f1e14db"}
Sep 11 18:15:03.885575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:03.885519    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="843255bd28a6aefbbf1e7bf1c38effeee577d3c8da7d95ba009b90253f1e14db"
Sep 11 18:15:04.094885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.094491    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl"
Sep 11 18:15:04.129635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.129608    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"
Sep 11 18:15:04.180224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.178425    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"
Sep 11 18:15:04.333235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333199    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"]
Sep 11 18:15:04.333721 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333707    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43732a78-29e3-40bb-ac1c-b30272c559ca" containerName="cleanup"
Sep 11 18:15:04.333721 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333722    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="43732a78-29e3-40bb-ac1c-b30272c559ca" containerName="cleanup"
Sep 11 18:15:04.333815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333729    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b5f6bf-694f-4238-91bc-c0efc6492347" containerName="cleanup"
Sep 11 18:15:04.333815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333735    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b5f6bf-694f-4238-91bc-c0efc6492347" containerName="cleanup"
Sep 11 18:15:04.333815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333746    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b6f835f-cb9d-4356-a4bc-7b3c03099802" containerName="cleanup"
Sep 11 18:15:04.333815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333752    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b6f835f-cb9d-4356-a4bc-7b3c03099802" containerName="cleanup"
Sep 11 18:15:04.333815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333758    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c044c58-ecd9-4ba2-a757-029e6975cfae" containerName="cleanup"
Sep 11 18:15:04.333815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333763    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c044c58-ecd9-4ba2-a757-029e6975cfae" containerName="cleanup"
Sep 11 18:15:04.333815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333776    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d" containerName="cleanup"
Sep 11 18:15:04.333815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333782    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d" containerName="cleanup"
Sep 11 18:15:04.334060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333854    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="43732a78-29e3-40bb-ac1c-b30272c559ca" containerName="cleanup"
Sep 11 18:15:04.334060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333862    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b5f6bf-694f-4238-91bc-c0efc6492347" containerName="cleanup"
Sep 11 18:15:04.334060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333868    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d" containerName="cleanup"
Sep 11 18:15:04.334060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333875    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c044c58-ecd9-4ba2-a757-029e6975cfae" containerName="cleanup"
Sep 11 18:15:04.334060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.333880    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b6f835f-cb9d-4356-a4bc-7b3c03099802" containerName="cleanup"
Sep 11 18:15:04.337004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.336988    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"
Sep 11 18:15:04.346601 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.346579    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"]
Sep 11 18:15:04.382952 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.382904    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520472d-4a5f-4e35-9b17-01e383bb73c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn\" (UID: \"7520472d-4a5f-4e35-9b17-01e383bb73c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"
Sep 11 18:15:04.483580 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.483539    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520472d-4a5f-4e35-9b17-01e383bb73c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn\" (UID: \"7520472d-4a5f-4e35-9b17-01e383bb73c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"
Sep 11 18:15:04.484149 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.484132    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520472d-4a5f-4e35-9b17-01e383bb73c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn\" (UID: \"7520472d-4a5f-4e35-9b17-01e383bb73c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"
Sep 11 18:15:04.652867 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.652834    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"
Sep 11 18:15:04.894900 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.894820    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"
Sep 11 18:15:04.894900 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.894866    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl" event={"ID":"43732a78-29e3-40bb-ac1c-b30272c559ca","Type":"ContainerDied","Data":"9c11ee14efb3e108aff3cebec2dc3849b931343df098c510f92f28b400f6159d"}
Sep 11 18:15:04.895354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.894916    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c11ee14efb3e108aff3cebec2dc3849b931343df098c510f92f28b400f6159d"
Sep 11 18:15:04.896348 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.896323    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq" event={"ID":"76b5f6bf-694f-4238-91bc-c0efc6492347","Type":"ContainerDied","Data":"8f66a3c52868fdb3813b5ce0f78925e6e4f47fd470c67ee82dda835447936598"}
Sep 11 18:15:04.896474 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.896372    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f66a3c52868fdb3813b5ce0f78925e6e4f47fd470c67ee82dda835447936598"
Sep 11 18:15:04.896474 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.896335    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"
Sep 11 18:15:04.897840 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.897815    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl"
Sep 11 18:15:04.897944 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.897825    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819175-j5lwl" event={"ID":"4c044c58-ecd9-4ba2-a757-029e6975cfae","Type":"ContainerDied","Data":"7acb44cc4b0482ad03c1fe735f4e5757c936bafe6c314cc9f7f25d8024ea5db7"}
Sep 11 18:15:04.897944 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.897928    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7acb44cc4b0482ad03c1fe735f4e5757c936bafe6c314cc9f7f25d8024ea5db7"
Sep 11 18:15:04.957066 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:04.957033    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"]
Sep 11 18:15:05.905673 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:05.905634    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn" event={"ID":"7520472d-4a5f-4e35-9b17-01e383bb73c2","Type":"ContainerStarted","Data":"e775e5607a9c13ca75f11a7e8569a43215e0aa8ea6b6ddce5d8920dab36f8a29"}
Sep 11 18:15:05.906063 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:05.905696    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn" event={"ID":"7520472d-4a5f-4e35-9b17-01e383bb73c2","Type":"ContainerStarted","Data":"52f7693d127d7c2f73947bc9ea19661f8d856dfe91ae05d40e750d872def0a10"}
Sep 11 18:15:05.928421 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:05.928374    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn" podStartSLOduration=1.928357367 podStartE2EDuration="1.928357367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:05.926053561 +0000 UTC m=+1269.696119921" watchObservedRunningTime="2026-09-11 18:15:05.928357367 +0000 UTC m=+1269.698423712"
Sep 11 18:15:06.913912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:06.913874    2571 generic.go:356] "Generic (PLEG): container finished" podID="7520472d-4a5f-4e35-9b17-01e383bb73c2" containerID="e775e5607a9c13ca75f11a7e8569a43215e0aa8ea6b6ddce5d8920dab36f8a29" exitCode=0
Sep 11 18:15:06.914268 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:06.913969    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn" event={"ID":"7520472d-4a5f-4e35-9b17-01e383bb73c2","Type":"ContainerDied","Data":"e775e5607a9c13ca75f11a7e8569a43215e0aa8ea6b6ddce5d8920dab36f8a29"}
Sep 11 18:15:08.118665 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:08.118642    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"
Sep 11 18:15:08.219520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:08.219430    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520472d-4a5f-4e35-9b17-01e383bb73c2-maas-api-service-ca\") pod \"7520472d-4a5f-4e35-9b17-01e383bb73c2\" (UID: \"7520472d-4a5f-4e35-9b17-01e383bb73c2\") "
Sep 11 18:15:08.219788 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:08.219765    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7520472d-4a5f-4e35-9b17-01e383bb73c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7520472d-4a5f-4e35-9b17-01e383bb73c2" (UID: "7520472d-4a5f-4e35-9b17-01e383bb73c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:08.320374 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:08.320343    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7520472d-4a5f-4e35-9b17-01e383bb73c2-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:08.928385 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:08.928356    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"
Sep 11 18:15:08.928571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:08.928353    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn" event={"ID":"7520472d-4a5f-4e35-9b17-01e383bb73c2","Type":"ContainerDied","Data":"52f7693d127d7c2f73947bc9ea19661f8d856dfe91ae05d40e750d872def0a10"}
Sep 11 18:15:08.928571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:08.928488    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52f7693d127d7c2f73947bc9ea19661f8d856dfe91ae05d40e750d872def0a10"
Sep 11 18:15:12.965520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:12.965484    2571 generic.go:356] "Generic (PLEG): container finished" podID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerID="2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8" exitCode=0
Sep 11 18:15:12.966143 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:12.965535    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" event={"ID":"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd","Type":"ContainerDied","Data":"2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8"}
Sep 11 18:15:12.966143 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:12.965912    2571 scope.go:117] "RemoveContainer" containerID="2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8"
Sep 11 18:15:12.967089 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:12.967058    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:15:12.967192 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:12.967148    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:15:12.967257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:12.967199    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:15:13.992764 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:13.992733    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"]
Sep 11 18:15:13.993229 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:13.992937    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w" podUID="049ad97c-9c3e-474b-a7bb-bc47605b6964" containerName="istio-proxy" containerID="cri-o://13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f" gracePeriod=30
Sep 11 18:15:13.999684 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:13.999661    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" event={"ID":"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd","Type":"ContainerStarted","Data":"3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8"}
Sep 11 18:15:14.509514 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:14.509479    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:15:14.509686 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:14.509647    2571 patch_prober.go:28] interesting pod/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 11 18:15:14.509731 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:14.509690    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 11 18:15:15.509821 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:15.509775    2571 patch_prober.go:28] interesting pod/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 11 18:15:15.510266 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:15.509854    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 11 18:15:16.510017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:16.509982    2571 patch_prober.go:28] interesting pod/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 11 18:15:16.510423 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:16.510046    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 11 18:15:17.509728 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:17.509692    2571 patch_prober.go:28] interesting pod/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 11 18:15:17.509903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:17.509751    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 11 18:15:17.789446 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:17.789352    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"]
Sep 11 18:15:18.509455 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:18.509418    2571 patch_prober.go:28] interesting pod/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 11 18:15:18.509635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:18.509477    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 11 18:15:19.291135 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.291108    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:15:19.333929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.333890    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"]
Sep 11 18:15:19.342373 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342346    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-token\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342533 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342393    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-data\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342533 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342426    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-certs\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342533 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342460    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-socket\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342533 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342502    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-credential-socket\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342536    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx4bl\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-kube-api-access-mx4bl\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342581    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-podinfo\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342608    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-envoy\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342681    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049ad97c-9c3e-474b-a7bb-bc47605b6964-istiod-ca-cert\") pod \"049ad97c-9c3e-474b-a7bb-bc47605b6964\" (UID: \"049ad97c-9c3e-474b-a7bb-bc47605b6964\") "
Sep 11 18:15:19.342756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342725    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:19.343056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.342883    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:19.343056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.343025    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-data" (OuterVolumeSpecName: "istio-data") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:19.343163 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.343093    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.343163 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.343112    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.343492 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.343439    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:19.343839 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.343818    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049ad97c-9c3e-474b-a7bb-bc47605b6964-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:19.346034 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.346012    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-kube-api-access-mx4bl" (OuterVolumeSpecName: "kube-api-access-mx4bl") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "kube-api-access-mx4bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:19.346139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.346041    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-token" (OuterVolumeSpecName: "istio-token") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:19.346331 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.346285    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:19.346331 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.346314    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "049ad97c-9c3e-474b-a7bb-bc47605b6964" (UID: "049ad97c-9c3e-474b-a7bb-bc47605b6964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:15:19.444184 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.444150    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.444184 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.444182    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.444184 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.444192    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/049ad97c-9c3e-474b-a7bb-bc47605b6964-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.444184 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.444200    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.444493 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.444208    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.444493 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.444216    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/049ad97c-9c3e-474b-a7bb-bc47605b6964-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.444493 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.444225    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4bl\" (UniqueName: \"kubernetes.io/projected/049ad97c-9c3e-474b-a7bb-bc47605b6964-kube-api-access-mx4bl\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:19.568458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.568375    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:15:19.568748 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.568714    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:15:19.569865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.569840    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:15:19.612188 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.612158    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"]
Sep 11 18:15:19.612393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.612361    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx" podUID="9add17f0-91fc-48ff-86ef-7f025016b04e" containerName="payload-processing" containerID="cri-o://e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac" gracePeriod=30
Sep 11 18:15:19.632747 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.632715    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"]
Sep 11 18:15:19.632988 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.632960    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz" podUID="57dc7d24-87a4-41f0-a5bf-8afc71170e54" containerName="payload-pre-processing" containerID="cri-o://1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa" gracePeriod=30
Sep 11 18:15:19.901475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.901352    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"]
Sep 11 18:15:19.907481 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.907434    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-93598a-zwdkn"]
Sep 11 18:15:19.940285 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.940258    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:15:19.951195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:19.951170    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:15:20.038896 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.038861    2571 generic.go:356] "Generic (PLEG): container finished" podID="049ad97c-9c3e-474b-a7bb-bc47605b6964" containerID="13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f" exitCode=0
Sep 11 18:15:20.039061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.038912    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"
Sep 11 18:15:20.039061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.038950    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w" event={"ID":"049ad97c-9c3e-474b-a7bb-bc47605b6964","Type":"ContainerDied","Data":"13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f"}
Sep 11 18:15:20.039061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.039003    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w" event={"ID":"049ad97c-9c3e-474b-a7bb-bc47605b6964","Type":"ContainerDied","Data":"aeb088dc968990f41cd0deab3e99419eafff8d0ad90c15dd96df8467b518f436"}
Sep 11 18:15:20.039061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.039031    2571 scope.go:117] "RemoveContainer" containerID="13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f"
Sep 11 18:15:20.040405 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.040363    2571 generic.go:356] "Generic (PLEG): container finished" podID="57dc7d24-87a4-41f0-a5bf-8afc71170e54" containerID="1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa" exitCode=0
Sep 11 18:15:20.040521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.040411    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"
Sep 11 18:15:20.040521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.040408    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz" event={"ID":"57dc7d24-87a4-41f0-a5bf-8afc71170e54","Type":"ContainerDied","Data":"1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa"}
Sep 11 18:15:20.040644 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.040528    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz" event={"ID":"57dc7d24-87a4-41f0-a5bf-8afc71170e54","Type":"ContainerDied","Data":"e33d6a837458e770d3cda5d130c500610ffd535c8cd1c90545e8af767f4386b0"}
Sep 11 18:15:20.041912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.041889    2571 generic.go:356] "Generic (PLEG): container finished" podID="9add17f0-91fc-48ff-86ef-7f025016b04e" containerID="e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac" exitCode=0
Sep 11 18:15:20.042078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.041922    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"
Sep 11 18:15:20.042078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.041984    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx" event={"ID":"9add17f0-91fc-48ff-86ef-7f025016b04e","Type":"ContainerDied","Data":"e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac"}
Sep 11 18:15:20.042078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.042029    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx" event={"ID":"9add17f0-91fc-48ff-86ef-7f025016b04e","Type":"ContainerDied","Data":"3255267e2987a8eda1c15626df4ff340dd73a0c6da58b6f49ba4e5d2493bff8d"}
Sep 11 18:15:20.049523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.049491    2571 scope.go:117] "RemoveContainer" containerID="13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f"
Sep 11 18:15:20.049826 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:15:20.049805    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f\": container with ID starting with 13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f not found: ID does not exist" containerID="13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f"
Sep 11 18:15:20.049925 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.049836    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f"} err="failed to get container status \"13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f\": rpc error: code = NotFound desc = could not find container \"13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f\": container with ID starting with 13162fbfadf957c858b2dca689c927ca806069f52faa079a68d41f2576f2281f not found: ID does not exist"
Sep 11 18:15:20.049925 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.049857    2571 scope.go:117] "RemoveContainer" containerID="1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa"
Sep 11 18:15:20.052687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.052664    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpt2h\" (UniqueName: \"kubernetes.io/projected/57dc7d24-87a4-41f0-a5bf-8afc71170e54-kube-api-access-kpt2h\") pod \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\" (UID: \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\") "
Sep 11 18:15:20.052803 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.052780    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5tjc\" (UniqueName: \"kubernetes.io/projected/9add17f0-91fc-48ff-86ef-7f025016b04e-kube-api-access-n5tjc\") pod \"9add17f0-91fc-48ff-86ef-7f025016b04e\" (UID: \"9add17f0-91fc-48ff-86ef-7f025016b04e\") "
Sep 11 18:15:20.052866 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.052852    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9add17f0-91fc-48ff-86ef-7f025016b04e-plugins-config-volume\") pod \"9add17f0-91fc-48ff-86ef-7f025016b04e\" (UID: \"9add17f0-91fc-48ff-86ef-7f025016b04e\") "
Sep 11 18:15:20.052922 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.052882    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dc7d24-87a4-41f0-a5bf-8afc71170e54-plugins-config-volume\") pod \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\" (UID: \"57dc7d24-87a4-41f0-a5bf-8afc71170e54\") "
Sep 11 18:15:20.053547 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.053520    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57dc7d24-87a4-41f0-a5bf-8afc71170e54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57dc7d24-87a4-41f0-a5bf-8afc71170e54" (UID: "57dc7d24-87a4-41f0-a5bf-8afc71170e54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:20.053689 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.053665    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9add17f0-91fc-48ff-86ef-7f025016b04e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9add17f0-91fc-48ff-86ef-7f025016b04e" (UID: "9add17f0-91fc-48ff-86ef-7f025016b04e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:20.055195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.055168    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dc7d24-87a4-41f0-a5bf-8afc71170e54-kube-api-access-kpt2h" (OuterVolumeSpecName: "kube-api-access-kpt2h") pod "57dc7d24-87a4-41f0-a5bf-8afc71170e54" (UID: "57dc7d24-87a4-41f0-a5bf-8afc71170e54"). InnerVolumeSpecName "kube-api-access-kpt2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:20.055514 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.055492    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9add17f0-91fc-48ff-86ef-7f025016b04e-kube-api-access-n5tjc" (OuterVolumeSpecName: "kube-api-access-n5tjc") pod "9add17f0-91fc-48ff-86ef-7f025016b04e" (UID: "9add17f0-91fc-48ff-86ef-7f025016b04e"). InnerVolumeSpecName "kube-api-access-n5tjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:20.059707 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.059691    2571 scope.go:117] "RemoveContainer" containerID="1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa"
Sep 11 18:15:20.060018 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:15:20.059983    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa\": container with ID starting with 1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa not found: ID does not exist" containerID="1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa"
Sep 11 18:15:20.060116 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.060019    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa"} err="failed to get container status \"1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa\": rpc error: code = NotFound desc = could not find container \"1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa\": container with ID starting with 1436456cfdaed9972cd8fa02c5c04699b57ba8a25a0d05b96472b7af85ff72fa not found: ID does not exist"
Sep 11 18:15:20.060116 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.060036    2571 scope.go:117] "RemoveContainer" containerID="e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac"
Sep 11 18:15:20.068464 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.068445    2571 scope.go:117] "RemoveContainer" containerID="e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac"
Sep 11 18:15:20.068758 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:15:20.068733    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac\": container with ID starting with e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac not found: ID does not exist" containerID="e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac"
Sep 11 18:15:20.068814 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.068757    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac"} err="failed to get container status \"e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac\": rpc error: code = NotFound desc = could not find container \"e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac\": container with ID starting with e7cb13e56e804aa084ab8e3e081803593bbf49193dafcf16c3c9ba24303e88ac not found: ID does not exist"
Sep 11 18:15:20.084879 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.084857    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"]
Sep 11 18:15:20.089242 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.089197    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-cd668b-openshift-default-5ffd59f45d-xt47w"]
Sep 11 18:15:20.154353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.153872    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5tjc\" (UniqueName: \"kubernetes.io/projected/9add17f0-91fc-48ff-86ef-7f025016b04e-kube-api-access-n5tjc\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:20.154353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.153907    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9add17f0-91fc-48ff-86ef-7f025016b04e-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:20.154353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.153921    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57dc7d24-87a4-41f0-a5bf-8afc71170e54-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:20.154353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.153933    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpt2h\" (UniqueName: \"kubernetes.io/projected/57dc7d24-87a4-41f0-a5bf-8afc71170e54-kube-api-access-kpt2h\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:20.378160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.378128    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"]
Sep 11 18:15:20.383419 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.383396    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-93598a-7867756478-szjsz"]
Sep 11 18:15:20.398969 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.398942    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"]
Sep 11 18:15:20.403581 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.403559    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-93598a-86bc5c88c5-4wknx"]
Sep 11 18:15:20.731834 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.731802    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="049ad97c-9c3e-474b-a7bb-bc47605b6964" path="/var/lib/kubelet/pods/049ad97c-9c3e-474b-a7bb-bc47605b6964/volumes"
Sep 11 18:15:20.732183 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.732169    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57dc7d24-87a4-41f0-a5bf-8afc71170e54" path="/var/lib/kubelet/pods/57dc7d24-87a4-41f0-a5bf-8afc71170e54/volumes"
Sep 11 18:15:20.732510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.732497    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7520472d-4a5f-4e35-9b17-01e383bb73c2" path="/var/lib/kubelet/pods/7520472d-4a5f-4e35-9b17-01e383bb73c2/volumes"
Sep 11 18:15:20.733083 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:20.733069    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9add17f0-91fc-48ff-86ef-7f025016b04e" path="/var/lib/kubelet/pods/9add17f0-91fc-48ff-86ef-7f025016b04e/volumes"
Sep 11 18:15:27.709079 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709044    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"]
Sep 11 18:15:27.709602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709515    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7520472d-4a5f-4e35-9b17-01e383bb73c2" containerName="revoke-keys"
Sep 11 18:15:27.709602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709529    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7520472d-4a5f-4e35-9b17-01e383bb73c2" containerName="revoke-keys"
Sep 11 18:15:27.709602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709545    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9add17f0-91fc-48ff-86ef-7f025016b04e" containerName="payload-processing"
Sep 11 18:15:27.709602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709551    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9add17f0-91fc-48ff-86ef-7f025016b04e" containerName="payload-processing"
Sep 11 18:15:27.709602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709558    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57dc7d24-87a4-41f0-a5bf-8afc71170e54" containerName="payload-pre-processing"
Sep 11 18:15:27.709602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709564    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="57dc7d24-87a4-41f0-a5bf-8afc71170e54" containerName="payload-pre-processing"
Sep 11 18:15:27.709602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709583    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="049ad97c-9c3e-474b-a7bb-bc47605b6964" containerName="istio-proxy"
Sep 11 18:15:27.709602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709588    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="049ad97c-9c3e-474b-a7bb-bc47605b6964" containerName="istio-proxy"
Sep 11 18:15:27.709865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709649    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="049ad97c-9c3e-474b-a7bb-bc47605b6964" containerName="istio-proxy"
Sep 11 18:15:27.709865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709658    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7520472d-4a5f-4e35-9b17-01e383bb73c2" containerName="revoke-keys"
Sep 11 18:15:27.709865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709667    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9add17f0-91fc-48ff-86ef-7f025016b04e" containerName="payload-processing"
Sep 11 18:15:27.709865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.709673    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="57dc7d24-87a4-41f0-a5bf-8afc71170e54" containerName="payload-pre-processing"
Sep 11 18:15:27.712858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.712838    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.715998 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.715976    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"kube-root-ca.crt\""
Sep 11 18:15:27.717316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.717281    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"distinct-2-w0-f8a2d4-kserve-self-signed-certs\""
Sep 11 18:15:27.717428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.717336    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"openshift-service-ca.crt\""
Sep 11 18:15:27.717428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.717368    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"default-dockercfg-l4gw2\""
Sep 11 18:15:27.724784 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.724762    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"]
Sep 11 18:15:27.827077 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.827047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-home\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.827246 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.827090    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-model-cache\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.827246 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.827183    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tls-certs\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.827352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.827264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tmp-dir\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.827352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.827290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbrz\" (UniqueName: \"kubernetes.io/projected/b1a85d31-13ef-4db7-9722-63d1ebee6e57-kube-api-access-sgbrz\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.827421 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.827373    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-dshm\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.928568 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.928525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-model-cache\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.928764 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.928601    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tls-certs\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.928764 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.928668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tmp-dir\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.928764 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.928696    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbrz\" (UniqueName: \"kubernetes.io/projected/b1a85d31-13ef-4db7-9722-63d1ebee6e57-kube-api-access-sgbrz\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.928939 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.928775    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-dshm\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.928939 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.928806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-home\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.929063 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.928960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-model-cache\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.929063 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.928999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tmp-dir\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.929177 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.929118    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-home\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.930842 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.930822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-dshm\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.931098 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.931081    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tls-certs\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:27.938290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:27.938272    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbrz\" (UniqueName: \"kubernetes.io/projected/b1a85d31-13ef-4db7-9722-63d1ebee6e57-kube-api-access-sgbrz\") pod \"distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:28.024260 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:28.024179    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:28.326726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:28.326698    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"]
Sep 11 18:15:28.326950 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:15:28.326928    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1a85d31_13ef_4db7_9722_63d1ebee6e57.slice/crio-22dd269d82d643a4c1215f7f3f7fc17165487f715bd43c31fd1680e354c39fa6 WatchSource:0}: Error finding container 22dd269d82d643a4c1215f7f3f7fc17165487f715bd43c31fd1680e354c39fa6: Status 404 returned error can't find the container with id 22dd269d82d643a4c1215f7f3f7fc17165487f715bd43c31fd1680e354c39fa6
Sep 11 18:15:29.121791 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:29.121751    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r" event={"ID":"b1a85d31-13ef-4db7-9722-63d1ebee6e57","Type":"ContainerStarted","Data":"bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45"}
Sep 11 18:15:29.122168 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:29.121802    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r" event={"ID":"b1a85d31-13ef-4db7-9722-63d1ebee6e57","Type":"ContainerStarted","Data":"22dd269d82d643a4c1215f7f3f7fc17165487f715bd43c31fd1680e354c39fa6"}
Sep 11 18:15:29.147453 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:29.147409    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r" podStartSLOduration=2.147394436 podStartE2EDuration="2.147394436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:29.1463401 +0000 UTC m=+1292.916406448" watchObservedRunningTime="2026-09-11 18:15:29.147394436 +0000 UTC m=+1292.917460784"
Sep 11 18:15:32.809175 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:32.809106    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc" podUID="df2f26e5-3de0-4b9c-96a9-31a61ebd146f" containerName="main" containerID="cri-o://f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9" gracePeriod=45
Sep 11 18:15:33.066544 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.066479    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:15:33.148765 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.148731    2571 generic.go:356] "Generic (PLEG): container finished" podID="df2f26e5-3de0-4b9c-96a9-31a61ebd146f" containerID="f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9" exitCode=0
Sep 11 18:15:33.148972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.148786    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"
Sep 11 18:15:33.148972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.148826    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc" event={"ID":"df2f26e5-3de0-4b9c-96a9-31a61ebd146f","Type":"ContainerDied","Data":"f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9"}
Sep 11 18:15:33.148972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.148873    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc" event={"ID":"df2f26e5-3de0-4b9c-96a9-31a61ebd146f","Type":"ContainerDied","Data":"c480478c479464a4f45a96848ca8a14c6efca97da444e8d22f7b4b8861ab26d9"}
Sep 11 18:15:33.148972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.148896    2571 scope.go:117] "RemoveContainer" containerID="f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9"
Sep 11 18:15:33.159183 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.159163    2571 scope.go:117] "RemoveContainer" containerID="f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9"
Sep 11 18:15:33.159503 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:15:33.159482    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9\": container with ID starting with f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9 not found: ID does not exist" containerID="f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9"
Sep 11 18:15:33.159570 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.159511    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9"} err="failed to get container status \"f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9\": rpc error: code = NotFound desc = could not find container \"f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9\": container with ID starting with f4d03f3eea6d02b7fec13fc93ae86cea40e0aab3d945ffb88d13ae851d9452a9 not found: ID does not exist"
Sep 11 18:15:33.182936 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.182916    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-dshm\") pod \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") "
Sep 11 18:15:33.183042 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.182954    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-home\") pod \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") "
Sep 11 18:15:33.183042 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.182984    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-model-cache\") pod \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") "
Sep 11 18:15:33.183042 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.183012    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tls-certs\") pod \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") "
Sep 11 18:15:33.183042 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.183033    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dfgz\" (UniqueName: \"kubernetes.io/projected/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-kube-api-access-4dfgz\") pod \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") "
Sep 11 18:15:33.183235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.183162    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tmp-dir\") pod \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\" (UID: \"df2f26e5-3de0-4b9c-96a9-31a61ebd146f\") "
Sep 11 18:15:33.183235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.183163    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-home" (OuterVolumeSpecName: "home") pod "df2f26e5-3de0-4b9c-96a9-31a61ebd146f" (UID: "df2f26e5-3de0-4b9c-96a9-31a61ebd146f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:33.183235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.183217    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-model-cache" (OuterVolumeSpecName: "model-cache") pod "df2f26e5-3de0-4b9c-96a9-31a61ebd146f" (UID: "df2f26e5-3de0-4b9c-96a9-31a61ebd146f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:33.183511 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.183485    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:33.183511 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.183507    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:33.183630 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.183480    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df2f26e5-3de0-4b9c-96a9-31a61ebd146f" (UID: "df2f26e5-3de0-4b9c-96a9-31a61ebd146f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:33.185101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.185076    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-dshm" (OuterVolumeSpecName: "dshm") pod "df2f26e5-3de0-4b9c-96a9-31a61ebd146f" (UID: "df2f26e5-3de0-4b9c-96a9-31a61ebd146f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:33.185207 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.185132    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df2f26e5-3de0-4b9c-96a9-31a61ebd146f" (UID: "df2f26e5-3de0-4b9c-96a9-31a61ebd146f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:33.185207 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.185174    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-kube-api-access-4dfgz" (OuterVolumeSpecName: "kube-api-access-4dfgz") pod "df2f26e5-3de0-4b9c-96a9-31a61ebd146f" (UID: "df2f26e5-3de0-4b9c-96a9-31a61ebd146f"). InnerVolumeSpecName "kube-api-access-4dfgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:33.284596 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.284548    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:33.284596 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.284589    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dfgz\" (UniqueName: \"kubernetes.io/projected/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-kube-api-access-4dfgz\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:33.284596 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.284600    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:33.284596 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.284610    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2f26e5-3de0-4b9c-96a9-31a61ebd146f-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:33.475139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.475106    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"]
Sep 11 18:15:33.480859 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:33.480834    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-93598a/test-model-93598a-kserve-5c598875f-vgqrc"]
Sep 11 18:15:34.361216 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.361142    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8" podUID="5a04df8d-d218-47b4-a898-6ccb92c32246" containerName="main" containerID="cri-o://9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e" gracePeriod=45
Sep 11 18:15:34.617889 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.617825    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:15:34.697925 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.697888    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a04df8d-d218-47b4-a898-6ccb92c32246-tls-certs\") pod \"5a04df8d-d218-47b4-a898-6ccb92c32246\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") "
Sep 11 18:15:34.698134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.697957    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-home\") pod \"5a04df8d-d218-47b4-a898-6ccb92c32246\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") "
Sep 11 18:15:34.698134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698030    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-tmp-dir\") pod \"5a04df8d-d218-47b4-a898-6ccb92c32246\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") "
Sep 11 18:15:34.698134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698059    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-model-cache\") pod \"5a04df8d-d218-47b4-a898-6ccb92c32246\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") "
Sep 11 18:15:34.698134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698103    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssxp7\" (UniqueName: \"kubernetes.io/projected/5a04df8d-d218-47b4-a898-6ccb92c32246-kube-api-access-ssxp7\") pod \"5a04df8d-d218-47b4-a898-6ccb92c32246\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") "
Sep 11 18:15:34.698376 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698217    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-dshm\") pod \"5a04df8d-d218-47b4-a898-6ccb92c32246\" (UID: \"5a04df8d-d218-47b4-a898-6ccb92c32246\") "
Sep 11 18:15:34.698376 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698217    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-home" (OuterVolumeSpecName: "home") pod "5a04df8d-d218-47b4-a898-6ccb92c32246" (UID: "5a04df8d-d218-47b4-a898-6ccb92c32246"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:34.698376 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698328    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a04df8d-d218-47b4-a898-6ccb92c32246" (UID: "5a04df8d-d218-47b4-a898-6ccb92c32246"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:34.698376 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698360    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a04df8d-d218-47b4-a898-6ccb92c32246" (UID: "5a04df8d-d218-47b4-a898-6ccb92c32246"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:34.698637 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698617    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:34.698675 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698666    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:34.698721 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.698682    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:34.700255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.700234    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-dshm" (OuterVolumeSpecName: "dshm") pod "5a04df8d-d218-47b4-a898-6ccb92c32246" (UID: "5a04df8d-d218-47b4-a898-6ccb92c32246"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:34.700361 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.700290    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a04df8d-d218-47b4-a898-6ccb92c32246-kube-api-access-ssxp7" (OuterVolumeSpecName: "kube-api-access-ssxp7") pod "5a04df8d-d218-47b4-a898-6ccb92c32246" (UID: "5a04df8d-d218-47b4-a898-6ccb92c32246"). InnerVolumeSpecName "kube-api-access-ssxp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:34.700361 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.700319    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a04df8d-d218-47b4-a898-6ccb92c32246-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a04df8d-d218-47b4-a898-6ccb92c32246" (UID: "5a04df8d-d218-47b4-a898-6ccb92c32246"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:34.731555 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.731521    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df2f26e5-3de0-4b9c-96a9-31a61ebd146f" path="/var/lib/kubelet/pods/df2f26e5-3de0-4b9c-96a9-31a61ebd146f/volumes"
Sep 11 18:15:34.799551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.799517    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssxp7\" (UniqueName: \"kubernetes.io/projected/5a04df8d-d218-47b4-a898-6ccb92c32246-kube-api-access-ssxp7\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:34.799551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.799546    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a04df8d-d218-47b4-a898-6ccb92c32246-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:34.799551 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:34.799556    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a04df8d-d218-47b4-a898-6ccb92c32246-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:35.167403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.165965    2571 generic.go:356] "Generic (PLEG): container finished" podID="5a04df8d-d218-47b4-a898-6ccb92c32246" containerID="9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e" exitCode=0
Sep 11 18:15:35.167403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.166091    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8" event={"ID":"5a04df8d-d218-47b4-a898-6ccb92c32246","Type":"ContainerDied","Data":"9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e"}
Sep 11 18:15:35.167403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.166137    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8" event={"ID":"5a04df8d-d218-47b4-a898-6ccb92c32246","Type":"ContainerDied","Data":"21844b91862754e4fa67bc912ef25940591093662a7d58de84293d168bab0c9e"}
Sep 11 18:15:35.167403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.166159    2571 scope.go:117] "RemoveContainer" containerID="9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e"
Sep 11 18:15:35.167403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.166280    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"
Sep 11 18:15:35.178185 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.178168    2571 scope.go:117] "RemoveContainer" containerID="9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e"
Sep 11 18:15:35.178468 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:15:35.178449    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e\": container with ID starting with 9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e not found: ID does not exist" containerID="9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e"
Sep 11 18:15:35.178554 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.178478    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e"} err="failed to get container status \"9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e\": rpc error: code = NotFound desc = could not find container \"9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e\": container with ID starting with 9bfa46cdf3d1c7582d40d469c1b3e046ccf40731b2e1cbe1f3189bd412aca90e not found: ID does not exist"
Sep 11 18:15:35.188129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.188106    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"]
Sep 11 18:15:35.192540 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:35.192519    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-cd668b/noten-model-cd668b-kserve-8c5c8647f-r88c8"]
Sep 11 18:15:36.734237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:36.734192    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a04df8d-d218-47b4-a898-6ccb92c32246" path="/var/lib/kubelet/pods/5a04df8d-d218-47b4-a898-6ccb92c32246/volumes"
Sep 11 18:15:38.024499 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:38.024470    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:38.024895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:38.024536    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:38.037410 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:38.037384    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:38.198368 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:38.198339    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:15:41.638071 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.638040    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"]
Sep 11 18:15:41.638469 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.638261    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy" containerID="cri-o://a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624" gracePeriod=30
Sep 11 18:15:41.995237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.995202    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"]
Sep 11 18:15:41.995653 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.995641    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a04df8d-d218-47b4-a898-6ccb92c32246" containerName="main"
Sep 11 18:15:41.995710 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.995655    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a04df8d-d218-47b4-a898-6ccb92c32246" containerName="main"
Sep 11 18:15:41.995710 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.995667    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df2f26e5-3de0-4b9c-96a9-31a61ebd146f" containerName="main"
Sep 11 18:15:41.995710 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.995673    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="df2f26e5-3de0-4b9c-96a9-31a61ebd146f" containerName="main"
Sep 11 18:15:41.995807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.995750    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="df2f26e5-3de0-4b9c-96a9-31a61ebd146f" containerName="main"
Sep 11 18:15:41.995807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:41.995762    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a04df8d-d218-47b4-a898-6ccb92c32246" containerName="main"
Sep 11 18:15:42.000450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:42.000430    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"
Sep 11 18:15:42.004003 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:42.003986    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:15:42.009287 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:42.009262    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"]
Sep 11 18:15:42.169723 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:42.169686    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0359f5-1788-4f6f-bbc4-a9cd474efe6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj\" (UID: \"2f0359f5-1788-4f6f-bbc4-a9cd474efe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"
Sep 11 18:15:42.271232 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:42.271137    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0359f5-1788-4f6f-bbc4-a9cd474efe6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj\" (UID: \"2f0359f5-1788-4f6f-bbc4-a9cd474efe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"
Sep 11 18:15:42.271846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:42.271822    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0359f5-1788-4f6f-bbc4-a9cd474efe6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj\" (UID: \"2f0359f5-1788-4f6f-bbc4-a9cd474efe6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"
Sep 11 18:15:42.313170 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:42.313125    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"
Sep 11 18:15:42.594728 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:42.594694    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"]
Sep 11 18:15:42.594949 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:15:42.594921    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0359f5_1788_4f6f_bbc4_a9cd474efe6f.slice/crio-fbdee014db5a96922566d5289a0beaee164e5f4a542a27233299606b6acc952e WatchSource:0}: Error finding container fbdee014db5a96922566d5289a0beaee164e5f4a542a27233299606b6acc952e: Status 404 returned error can't find the container with id fbdee014db5a96922566d5289a0beaee164e5f4a542a27233299606b6acc952e
Sep 11 18:15:43.220420 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:43.220383    2571 generic.go:356] "Generic (PLEG): container finished" podID="2f0359f5-1788-4f6f-bbc4-a9cd474efe6f" containerID="9cfbd6f1db754609f8696dcb657ca6d833023727ddad3e25650843d354fe70bb" exitCode=0
Sep 11 18:15:43.220780 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:43.220468    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj" event={"ID":"2f0359f5-1788-4f6f-bbc4-a9cd474efe6f","Type":"ContainerDied","Data":"9cfbd6f1db754609f8696dcb657ca6d833023727ddad3e25650843d354fe70bb"}
Sep 11 18:15:43.220780 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:43.220532    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj" event={"ID":"2f0359f5-1788-4f6f-bbc4-a9cd474efe6f","Type":"ContainerStarted","Data":"fbdee014db5a96922566d5289a0beaee164e5f4a542a27233299606b6acc952e"}
Sep 11 18:15:44.361607 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:44.361582    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"
Sep 11 18:15:44.389644 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:44.389617    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0359f5-1788-4f6f-bbc4-a9cd474efe6f-maas-api-service-ca\") pod \"2f0359f5-1788-4f6f-bbc4-a9cd474efe6f\" (UID: \"2f0359f5-1788-4f6f-bbc4-a9cd474efe6f\") "
Sep 11 18:15:44.389926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:44.389902    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0359f5-1788-4f6f-bbc4-a9cd474efe6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f0359f5-1788-4f6f-bbc4-a9cd474efe6f" (UID: "2f0359f5-1788-4f6f-bbc4-a9cd474efe6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:44.390021 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:44.389966    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f0359f5-1788-4f6f-bbc4-a9cd474efe6f-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:45.235002 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:45.234970    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj" event={"ID":"2f0359f5-1788-4f6f-bbc4-a9cd474efe6f","Type":"ContainerDied","Data":"fbdee014db5a96922566d5289a0beaee164e5f4a542a27233299606b6acc952e"}
Sep 11 18:15:45.235209 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:45.235008    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"
Sep 11 18:15:45.235209 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:45.235019    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbdee014db5a96922566d5289a0beaee164e5f4a542a27233299606b6acc952e"
Sep 11 18:15:46.901094 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.901037    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:15:46.915491 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915464    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f29dfd-097e-4f99-afe8-c004429331af-istio-podinfo\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.915628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915516    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-credential-socket\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.915628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915563    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-envoy\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.915628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915603    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-socket\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.915807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915637    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f29dfd-097e-4f99-afe8-c004429331af-istiod-ca-cert\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.915807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915666    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-data\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.915807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915701    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-certs\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.915807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915750    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpjrv\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-kube-api-access-cpjrv\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.915807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915782    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-istio-token\") pod \"69f29dfd-097e-4f99-afe8-c004429331af\" (UID: \"69f29dfd-097e-4f99-afe8-c004429331af\") "
Sep 11 18:15:46.916073 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915816    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:46.916073 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:46.916073 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.915997    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-data" (OuterVolumeSpecName: "istio-data") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:46.916073 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.916013    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f29dfd-097e-4f99-afe8-c004429331af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:46.916073 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.916027    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:46.916351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.916188    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:46.916351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.916209    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f29dfd-097e-4f99-afe8-c004429331af-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:46.916351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.916224    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:46.916351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.916237    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:46.916351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.916250    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:46.917691 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.917667    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69f29dfd-097e-4f99-afe8-c004429331af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:15:46.917826 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.917804    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-kube-api-access-cpjrv" (OuterVolumeSpecName: "kube-api-access-cpjrv") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "kube-api-access-cpjrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:46.918064 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.918049    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:46.918139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:46.918125    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-istio-token" (OuterVolumeSpecName: "istio-token") pod "69f29dfd-097e-4f99-afe8-c004429331af" (UID: "69f29dfd-097e-4f99-afe8-c004429331af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:47.016880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.016849    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f29dfd-097e-4f99-afe8-c004429331af-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.016880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.016872    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f29dfd-097e-4f99-afe8-c004429331af-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.016880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.016882    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpjrv\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-kube-api-access-cpjrv\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.016880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.016890    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f29dfd-097e-4f99-afe8-c004429331af-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.250777 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.250739    2571 generic.go:356] "Generic (PLEG): container finished" podID="69f29dfd-097e-4f99-afe8-c004429331af" containerID="a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624" exitCode=0
Sep 11 18:15:47.250777 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.250769    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" event={"ID":"69f29dfd-097e-4f99-afe8-c004429331af","Type":"ContainerDied","Data":"a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624"}
Sep 11 18:15:47.250984 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.250829    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42" event={"ID":"69f29dfd-097e-4f99-afe8-c004429331af","Type":"ContainerDied","Data":"09c296bf8695fb8826f45dc9f98ad7475bf82b5a98e5044a57afd9be9a114187"}
Sep 11 18:15:47.250984 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.250781    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"
Sep 11 18:15:47.250984 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.250855    2571 scope.go:117] "RemoveContainer" containerID="a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624"
Sep 11 18:15:47.269018 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.268995    2571 scope.go:117] "RemoveContainer" containerID="d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6"
Sep 11 18:15:47.281310 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.281282    2571 scope.go:117] "RemoveContainer" containerID="a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624"
Sep 11 18:15:47.281612 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:15:47.281595    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624\": container with ID starting with a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624 not found: ID does not exist" containerID="a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624"
Sep 11 18:15:47.281654 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.281619    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624"} err="failed to get container status \"a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624\": rpc error: code = NotFound desc = could not find container \"a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624\": container with ID starting with a11b86877db8e3dcbd6fb2c48230e1d25db6f901fc0df34721903190dabda624 not found: ID does not exist"
Sep 11 18:15:47.281654 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.281640    2571 scope.go:117] "RemoveContainer" containerID="d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6"
Sep 11 18:15:47.281909 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:15:47.281889    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6\": container with ID starting with d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6 not found: ID does not exist" containerID="d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6"
Sep 11 18:15:47.281972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.281927    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6"} err="failed to get container status \"d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6\": rpc error: code = NotFound desc = could not find container \"d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6\": container with ID starting with d1a76e719634d1562084f5e254dc7f96665d9dec6b038524e9a267757d3c3ed6 not found: ID does not exist"
Sep 11 18:15:47.303077 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.303046    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"]
Sep 11 18:15:47.313230 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:47.313207    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-93598a-openshift-default-7cd7f95d76-hsd42"]
Sep 11 18:15:48.731287 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:48.731255    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f29dfd-097e-4f99-afe8-c004429331af" path="/var/lib/kubelet/pods/69f29dfd-097e-4f99-afe8-c004429331af/volumes"
Sep 11 18:15:52.617472 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.617432    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"]
Sep 11 18:15:52.617954 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.617679    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq" podUID="ecf028d7-ebfa-4768-8728-96adb66e7b7d" containerName="payload-pre-processing" containerID="cri-o://e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765" gracePeriod=30
Sep 11 18:15:52.809903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.809868    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"]
Sep 11 18:15:52.814461 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.814429    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d59a02-5zwlj"]
Sep 11 18:15:52.881400 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.881343    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:15:52.965340 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.965280    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf028d7-ebfa-4768-8728-96adb66e7b7d-plugins-config-volume\") pod \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\" (UID: \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\") "
Sep 11 18:15:52.965517 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.965445    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j8gh\" (UniqueName: \"kubernetes.io/projected/ecf028d7-ebfa-4768-8728-96adb66e7b7d-kube-api-access-8j8gh\") pod \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\" (UID: \"ecf028d7-ebfa-4768-8728-96adb66e7b7d\") "
Sep 11 18:15:52.965708 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.965684    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf028d7-ebfa-4768-8728-96adb66e7b7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecf028d7-ebfa-4768-8728-96adb66e7b7d" (UID: "ecf028d7-ebfa-4768-8728-96adb66e7b7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:52.968451 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:52.968419    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf028d7-ebfa-4768-8728-96adb66e7b7d-kube-api-access-8j8gh" (OuterVolumeSpecName: "kube-api-access-8j8gh") pod "ecf028d7-ebfa-4768-8728-96adb66e7b7d" (UID: "ecf028d7-ebfa-4768-8728-96adb66e7b7d"). InnerVolumeSpecName "kube-api-access-8j8gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:53.066364 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.066322    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf028d7-ebfa-4768-8728-96adb66e7b7d-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:53.066364 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.066353    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8gh\" (UniqueName: \"kubernetes.io/projected/ecf028d7-ebfa-4768-8728-96adb66e7b7d-kube-api-access-8j8gh\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:15:53.295627 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.295594    2571 generic.go:356] "Generic (PLEG): container finished" podID="ecf028d7-ebfa-4768-8728-96adb66e7b7d" containerID="e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765" exitCode=0
Sep 11 18:15:53.295816 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.295639    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"
Sep 11 18:15:53.295816 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.295680    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq" event={"ID":"ecf028d7-ebfa-4768-8728-96adb66e7b7d","Type":"ContainerDied","Data":"e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765"}
Sep 11 18:15:53.295816 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.295730    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq" event={"ID":"ecf028d7-ebfa-4768-8728-96adb66e7b7d","Type":"ContainerDied","Data":"7750f77cd5630849e0ef7ab541e4f8fe66ce40409b23273bcf9b02ef2a9e7d2d"}
Sep 11 18:15:53.295816 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.295753    2571 scope.go:117] "RemoveContainer" containerID="e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765"
Sep 11 18:15:53.306155 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.306128    2571 scope.go:117] "RemoveContainer" containerID="e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765"
Sep 11 18:15:53.306467 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:15:53.306444    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765\": container with ID starting with e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765 not found: ID does not exist" containerID="e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765"
Sep 11 18:15:53.306519 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.306479    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765"} err="failed to get container status \"e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765\": rpc error: code = NotFound desc = could not find container \"e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765\": container with ID starting with e7f577751132bd23c32898491aa9f0af7cd23028176c8bba6aea122176536765 not found: ID does not exist"
Sep 11 18:15:53.320564 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.320406    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"]
Sep 11 18:15:53.323773 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:53.323753    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d59a02-8f778cd-zhrsq"]
Sep 11 18:15:54.731606 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:54.731568    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f0359f5-1788-4f6f-bbc4-a9cd474efe6f" path="/var/lib/kubelet/pods/2f0359f5-1788-4f6f-bbc4-a9cd474efe6f/volumes"
Sep 11 18:15:54.732075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:54.732052    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf028d7-ebfa-4768-8728-96adb66e7b7d" path="/var/lib/kubelet/pods/ecf028d7-ebfa-4768-8728-96adb66e7b7d/volumes"
Sep 11 18:15:56.992180 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:56.992143    2571 scope.go:117] "RemoveContainer" containerID="eeba4717353c4ad5d9eb11b3a2285b5e1be28661364b7ca47fe17bfbe7ea6273"
Sep 11 18:15:57.001384 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:15:57.001359    2571 scope.go:117] "RemoveContainer" containerID="54cc1113d998d327bcf4341dea812a4330fe381bb977bf2b98db93203fc8c9de"
Sep 11 18:16:00.122093 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:00.122060    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"]
Sep 11 18:16:00.122693 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:00.122381    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy" containerID="cri-o://ea47e7f2f60ecee507fad6171ed0b97b23385f92b3090ecb901d1f1a2aeb7166" gracePeriod=30
Sep 11 18:16:01.006793 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.006764    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"]
Sep 11 18:16:01.007211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007198    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy"
Sep 11 18:16:01.007211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007212    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007220    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf028d7-ebfa-4768-8728-96adb66e7b7d" containerName="payload-pre-processing"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007226    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf028d7-ebfa-4768-8728-96adb66e7b7d" containerName="payload-pre-processing"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007242    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007247    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007254    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f0359f5-1788-4f6f-bbc4-a9cd474efe6f" containerName="revoke-keys"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007260    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f0359f5-1788-4f6f-bbc4-a9cd474efe6f" containerName="revoke-keys"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007349    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007358    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f0359f5-1788-4f6f-bbc4-a9cd474efe6f" containerName="revoke-keys"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007363    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf028d7-ebfa-4768-8728-96adb66e7b7d" containerName="payload-pre-processing"
Sep 11 18:16:01.007382 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.007369    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f29dfd-097e-4f99-afe8-c004429331af" containerName="istio-proxy"
Sep 11 18:16:01.011664 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.011648    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.015439 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.015416    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a5bbf6-openshift-default-dockercfg-747s5\""
Sep 11 18:16:01.036409 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.036387    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"]
Sep 11 18:16:01.042875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.042853    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzgpt\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-kube-api-access-kzgpt\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.042979 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.042909    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-data\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.043038 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.043010    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-socket\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.043091 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.043065    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-credential-socket\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.043135 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.043119    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-envoy\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.043190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.043176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-token\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.043233 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.043223    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-certs\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.043275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.043248    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istiod-ca-cert\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.043353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.043314    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-podinfo\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.144520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-podinfo\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144553    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzgpt\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-kube-api-access-kzgpt\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144593    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-data\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144634    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-socket\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144660    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-credential-socket\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-envoy\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144718    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-token\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144845    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-certs\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.144888    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istiod-ca-cert\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.145188    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-socket\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.145308    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-certs\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.145345    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-data\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.145462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-credential-socket\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.145715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.145629    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istiod-ca-cert\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.147143 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.147123    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-envoy\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.147252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.147189    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-podinfo\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.162869 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.162843    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-token\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.164328 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.164283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzgpt\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-kube-api-access-kzgpt\") pod \"e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") " pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.321522 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.321435    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:01.604319 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.604271    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"]
Sep 11 18:16:01.605329 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:16:01.605290    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2557dc4d_7ea1_4bad_ac5a_44e21d0a4b0b.slice/crio-ca9e8dab9ea9096fd7677aced857548b75c5a7c791d420c4be3af2de1cbf1e9e WatchSource:0}: Error finding container ca9e8dab9ea9096fd7677aced857548b75c5a7c791d420c4be3af2de1cbf1e9e: Status 404 returned error can't find the container with id ca9e8dab9ea9096fd7677aced857548b75c5a7c791d420c4be3af2de1cbf1e9e
Sep 11 18:16:01.607580 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.607553    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:16:01.607656 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.607612    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:16:01.607656 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:01.607639    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:16:02.360545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:02.360501    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" event={"ID":"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b","Type":"ContainerStarted","Data":"7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f"}
Sep 11 18:16:02.360940 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:02.360563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" event={"ID":"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b","Type":"ContainerStarted","Data":"ca9e8dab9ea9096fd7677aced857548b75c5a7c791d420c4be3af2de1cbf1e9e"}
Sep 11 18:16:02.388858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:02.388813    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podStartSLOduration=2.388797844 podStartE2EDuration="2.388797844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:02.387010401 +0000 UTC m=+1326.157076749" watchObservedRunningTime="2026-09-11 18:16:02.388797844 +0000 UTC m=+1326.158864191"
Sep 11 18:16:03.322282 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:03.322247    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:03.327170 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:03.327145    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:03.367272 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:03.367242    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:03.368316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:03.368282    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:05.384194 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.384128    2571 generic.go:356] "Generic (PLEG): container finished" podID="c628866d-0242-49cd-b3ef-766ad525bfce" containerID="ea47e7f2f60ecee507fad6171ed0b97b23385f92b3090ecb901d1f1a2aeb7166" exitCode=0
Sep 11 18:16:05.384558 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.384201    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" event={"ID":"c628866d-0242-49cd-b3ef-766ad525bfce","Type":"ContainerDied","Data":"ea47e7f2f60ecee507fad6171ed0b97b23385f92b3090ecb901d1f1a2aeb7166"}
Sep 11 18:16:05.384558 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.384266    2571 scope.go:117] "RemoveContainer" containerID="61585775a3e7c365c7361d4eba735ac4674ad459e28ba33e0f30ec959707cda1"
Sep 11 18:16:05.400443 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.400424    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:16:05.484533 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484500    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-socket\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.484715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484563    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl9vj\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-kube-api-access-fl9vj\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.484715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484621    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-envoy\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.484715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484656    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-credential-socket\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.484715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484689    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c628866d-0242-49cd-b3ef-766ad525bfce-istiod-ca-cert\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.484941 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484718    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-istio-token\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.484941 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484761    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-data\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.484941 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484809    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-certs\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.484941 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:16:05.484941 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.484846    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c628866d-0242-49cd-b3ef-766ad525bfce-istio-podinfo\") pod \"c628866d-0242-49cd-b3ef-766ad525bfce\" (UID: \"c628866d-0242-49cd-b3ef-766ad525bfce\") "
Sep 11 18:16:05.485217 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.485002    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:16:05.485217 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.485089    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-data" (OuterVolumeSpecName: "istio-data") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:16:05.485217 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.485114    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:16:05.485432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.485339    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:05.485432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.485358    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:05.485432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.485373    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:05.485432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.485386    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:05.485724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.485703    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c628866d-0242-49cd-b3ef-766ad525bfce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:16:05.486929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.486902    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-kube-api-access-fl9vj" (OuterVolumeSpecName: "kube-api-access-fl9vj") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "kube-api-access-fl9vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:16:05.487023 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.486972    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c628866d-0242-49cd-b3ef-766ad525bfce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:16:05.487142 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.487126    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:16:05.487227 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.487214    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-istio-token" (OuterVolumeSpecName: "istio-token") pod "c628866d-0242-49cd-b3ef-766ad525bfce" (UID: "c628866d-0242-49cd-b3ef-766ad525bfce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:16:05.586434 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.586396    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c628866d-0242-49cd-b3ef-766ad525bfce-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:05.586434 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.586426    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:05.586434 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.586435    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c628866d-0242-49cd-b3ef-766ad525bfce-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:05.586666 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.586443    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl9vj\" (UniqueName: \"kubernetes.io/projected/c628866d-0242-49cd-b3ef-766ad525bfce-kube-api-access-fl9vj\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:05.586666 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:05.586454    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c628866d-0242-49cd-b3ef-766ad525bfce-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:16:06.392436 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:06.392409    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"
Sep 11 18:16:06.392850 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:06.392433    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt" event={"ID":"c628866d-0242-49cd-b3ef-766ad525bfce","Type":"ContainerDied","Data":"4c1452300c4964729cfdb243881afb2ac720a067f0a081514844748447afc933"}
Sep 11 18:16:06.392850 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:06.392486    2571 scope.go:117] "RemoveContainer" containerID="ea47e7f2f60ecee507fad6171ed0b97b23385f92b3090ecb901d1f1a2aeb7166"
Sep 11 18:16:06.441345 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:06.441286    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"]
Sep 11 18:16:06.443937 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:06.443914    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d59a02-openshift-default-6696b85c4-qvzqt"]
Sep 11 18:16:06.732741 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:06.732699    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" path="/var/lib/kubelet/pods/c628866d-0242-49cd-b3ef-766ad525bfce/volumes"
Sep 11 18:16:09.000909 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.000874    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"]
Sep 11 18:16:09.001467 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.001451    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy"
Sep 11 18:16:09.001522 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.001469    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy"
Sep 11 18:16:09.001565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.001550    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy"
Sep 11 18:16:09.006158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.006137    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.009642 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.009622    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3a652320\"/\"default-dockercfg-mm9gq\""
Sep 11 18:16:09.010960 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.010934    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3a652320\"/\"kube-root-ca.crt\""
Sep 11 18:16:09.010960 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.010951    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3a652320\"/\"e2e-lifecycle-model-3a652320-kserve-self-signed-certs\""
Sep 11 18:16:09.011099 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.011018    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3a652320\"/\"openshift-service-ca.crt\""
Sep 11 18:16:09.016717 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.016694    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"]
Sep 11 18:16:09.121799 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.121769    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-dshm\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.121968 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.121811    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqswr\" (UniqueName: \"kubernetes.io/projected/0802c482-a657-4fb1-ba1f-90a6038584b0-kube-api-access-nqswr\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.121968 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.121921    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0802c482-a657-4fb1-ba1f-90a6038584b0-tls-certs\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.121968 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.121946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-model-cache\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.121968 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.121962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-home\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.122111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.121976    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-tmp-dir\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223439 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0802c482-a657-4fb1-ba1f-90a6038584b0-tls-certs\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223587 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223448    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-model-cache\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223587 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223467    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-home\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223694 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-tmp-dir\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223694 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223665    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-dshm\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223785 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqswr\" (UniqueName: \"kubernetes.io/projected/0802c482-a657-4fb1-ba1f-90a6038584b0-kube-api-access-nqswr\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223785 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-home\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223826    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-model-cache\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.223959 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.223892    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-tmp-dir\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.225778 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.225754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-dshm\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.225893 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.225871    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0802c482-a657-4fb1-ba1f-90a6038584b0-tls-certs\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.233899 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.233868    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqswr\" (UniqueName: \"kubernetes.io/projected/0802c482-a657-4fb1-ba1f-90a6038584b0-kube-api-access-nqswr\") pod \"e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") " pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.317325 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.317235    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:09.599703 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:16:09.599673    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0802c482_a657_4fb1_ba1f_90a6038584b0.slice/crio-d5131b790f2a85d06aeea9842bfa9f9201daefaa1f0186596a0ed6c725669915 WatchSource:0}: Error finding container d5131b790f2a85d06aeea9842bfa9f9201daefaa1f0186596a0ed6c725669915: Status 404 returned error can't find the container with id d5131b790f2a85d06aeea9842bfa9f9201daefaa1f0186596a0ed6c725669915
Sep 11 18:16:09.599912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:09.599887    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"]
Sep 11 18:16:10.082142 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.082100    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"]
Sep 11 18:16:10.082602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.082586    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy"
Sep 11 18:16:10.082655 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.082604    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy"
Sep 11 18:16:10.082692 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.082684    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c628866d-0242-49cd-b3ef-766ad525bfce" containerName="istio-proxy"
Sep 11 18:16:10.085775 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.085758    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"]
Sep 11 18:16:10.085874 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.085861    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:10.088864 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.088835    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a5bbf6\""
Sep 11 18:16:10.089201 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.089185    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:10.098101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.098060    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"]
Sep 11 18:16:10.102600 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.102519    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"]
Sep 11 18:16:10.132102 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.132072    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmvg\" (UniqueName: \"kubernetes.io/projected/ac7a6760-f70c-431d-86fe-98f082668837-kube-api-access-qpmvg\") pod \"payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r\" (UID: \"ac7a6760-f70c-431d-86fe-98f082668837\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:10.132290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.132127    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxb6w\" (UniqueName: \"kubernetes.io/projected/73702daf-d255-4e06-8dd0-7956fd128508-kube-api-access-xxb6w\") pod \"payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956\" (UID: \"73702daf-d255-4e06-8dd0-7956fd128508\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:10.132290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.132227    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73702daf-d255-4e06-8dd0-7956fd128508-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956\" (UID: \"73702daf-d255-4e06-8dd0-7956fd128508\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:10.132290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.132251    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7a6760-f70c-431d-86fe-98f082668837-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r\" (UID: \"ac7a6760-f70c-431d-86fe-98f082668837\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:10.233477 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.233438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmvg\" (UniqueName: \"kubernetes.io/projected/ac7a6760-f70c-431d-86fe-98f082668837-kube-api-access-qpmvg\") pod \"payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r\" (UID: \"ac7a6760-f70c-431d-86fe-98f082668837\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:10.233777 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.233752    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxb6w\" (UniqueName: \"kubernetes.io/projected/73702daf-d255-4e06-8dd0-7956fd128508-kube-api-access-xxb6w\") pod \"payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956\" (UID: \"73702daf-d255-4e06-8dd0-7956fd128508\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:10.233985 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.233965    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73702daf-d255-4e06-8dd0-7956fd128508-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956\" (UID: \"73702daf-d255-4e06-8dd0-7956fd128508\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:10.234107 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.234092    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7a6760-f70c-431d-86fe-98f082668837-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r\" (UID: \"ac7a6760-f70c-431d-86fe-98f082668837\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:10.235055 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.235032    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7a6760-f70c-431d-86fe-98f082668837-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r\" (UID: \"ac7a6760-f70c-431d-86fe-98f082668837\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:10.235232 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.235040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73702daf-d255-4e06-8dd0-7956fd128508-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956\" (UID: \"73702daf-d255-4e06-8dd0-7956fd128508\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:10.243279 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.243253    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmvg\" (UniqueName: \"kubernetes.io/projected/ac7a6760-f70c-431d-86fe-98f082668837-kube-api-access-qpmvg\") pod \"payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r\" (UID: \"ac7a6760-f70c-431d-86fe-98f082668837\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:10.243549 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.243531    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxb6w\" (UniqueName: \"kubernetes.io/projected/73702daf-d255-4e06-8dd0-7956fd128508-kube-api-access-xxb6w\") pod \"payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956\" (UID: \"73702daf-d255-4e06-8dd0-7956fd128508\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:10.398473 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.398395    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:10.406559 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.406541    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:10.434883 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.434853    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4" event={"ID":"0802c482-a657-4fb1-ba1f-90a6038584b0","Type":"ContainerStarted","Data":"0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d"}
Sep 11 18:16:10.435041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.434894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4" event={"ID":"0802c482-a657-4fb1-ba1f-90a6038584b0","Type":"ContainerStarted","Data":"d5131b790f2a85d06aeea9842bfa9f9201daefaa1f0186596a0ed6c725669915"}
Sep 11 18:16:10.459875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.459828    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4" podStartSLOduration=2.459809167 podStartE2EDuration="2.459809167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:10.456099828 +0000 UTC m=+1334.226166177" watchObservedRunningTime="2026-09-11 18:16:10.459809167 +0000 UTC m=+1334.229875515"
Sep 11 18:16:10.684937 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:16:10.684909    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7a6760_f70c_431d_86fe_98f082668837.slice/crio-0c8a5fc1591675fdcd673f20d183206f8dc561cbe1d6b1e320bb31bd07ba7bc4 WatchSource:0}: Error finding container 0c8a5fc1591675fdcd673f20d183206f8dc561cbe1d6b1e320bb31bd07ba7bc4: Status 404 returned error can't find the container with id 0c8a5fc1591675fdcd673f20d183206f8dc561cbe1d6b1e320bb31bd07ba7bc4
Sep 11 18:16:10.685090 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.685065    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"]
Sep 11 18:16:10.699873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:10.699851    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"]
Sep 11 18:16:10.700884 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:16:10.700863    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73702daf_d255_4e06_8dd0_7956fd128508.slice/crio-1983cc12354c44fff87fd7ee535863938f8250d2d84baa9317168bc1dec648a3 WatchSource:0}: Error finding container 1983cc12354c44fff87fd7ee535863938f8250d2d84baa9317168bc1dec648a3: Status 404 returned error can't find the container with id 1983cc12354c44fff87fd7ee535863938f8250d2d84baa9317168bc1dec648a3
Sep 11 18:16:11.453865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:11.453799    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956" event={"ID":"73702daf-d255-4e06-8dd0-7956fd128508","Type":"ContainerStarted","Data":"714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f"}
Sep 11 18:16:11.454371 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:11.453907    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956" event={"ID":"73702daf-d255-4e06-8dd0-7956fd128508","Type":"ContainerStarted","Data":"1983cc12354c44fff87fd7ee535863938f8250d2d84baa9317168bc1dec648a3"}
Sep 11 18:16:11.454371 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:11.454274    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:11.455902 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:11.455878    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r" event={"ID":"ac7a6760-f70c-431d-86fe-98f082668837","Type":"ContainerStarted","Data":"5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5"}
Sep 11 18:16:11.456001 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:11.455917    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r" event={"ID":"ac7a6760-f70c-431d-86fe-98f082668837","Type":"ContainerStarted","Data":"0c8a5fc1591675fdcd673f20d183206f8dc561cbe1d6b1e320bb31bd07ba7bc4"}
Sep 11 18:16:11.478967 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:11.478919    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956" podStartSLOduration=1.478903502 podStartE2EDuration="1.478903502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:11.475558607 +0000 UTC m=+1335.245624966" watchObservedRunningTime="2026-09-11 18:16:11.478903502 +0000 UTC m=+1335.248969905"
Sep 11 18:16:11.498522 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:11.498467    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r" podStartSLOduration=1.498448926 podStartE2EDuration="1.498448926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:11.496425016 +0000 UTC m=+1335.266491365" watchObservedRunningTime="2026-09-11 18:16:11.498448926 +0000 UTC m=+1335.268515275"
Sep 11 18:16:12.463937 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:12.463906    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:19.317892 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:19.317800    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:19.318327 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:19.318118    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:19.330757 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:19.330732    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:19.521999 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:19.521971    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:16:22.466005 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:22.465973    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:16:23.473867 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:23.473834    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:16:23.954470 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:23.954435    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"]
Sep 11 18:16:23.959658 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:23.959635    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:23.966995 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:23.966971    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"openshift-service-ca.crt\""
Sep 11 18:16:23.967085 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:23.967027    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"default-dockercfg-fzqfp\""
Sep 11 18:16:23.968174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:23.968155    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"kube-root-ca.crt\""
Sep 11 18:16:23.968949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:23.968934    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"e2e-conflict-b-74f751c5-kserve-self-signed-certs\""
Sep 11 18:16:23.978868 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:23.978844    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"]
Sep 11 18:16:24.072620 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.072583    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tmp-dir\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.072809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.072628    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-model-cache\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.072809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.072694    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tdkb\" (UniqueName: \"kubernetes.io/projected/a19e41e3-b082-4b20-98dd-33cf3c50dc70-kube-api-access-2tdkb\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.072809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.072747    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-home\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.072809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.072782    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-dshm\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.072968 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.072811    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tls-certs\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174243    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tmp-dir\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-model-cache\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174336    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tdkb\" (UniqueName: \"kubernetes.io/projected/a19e41e3-b082-4b20-98dd-33cf3c50dc70-kube-api-access-2tdkb\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174364    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-home\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174387    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-dshm\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174409    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tls-certs\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174749 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174699    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tmp-dir\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174749 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174716    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-model-cache\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.174840 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.174768    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-home\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.176637 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.176618    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-dshm\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.176900 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.176883    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tls-certs\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.185514 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.185493    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tdkb\" (UniqueName: \"kubernetes.io/projected/a19e41e3-b082-4b20-98dd-33cf3c50dc70-kube-api-access-2tdkb\") pod \"e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.269879 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.269502    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:24.548953 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:24.548923    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"]
Sep 11 18:16:24.551013 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:16:24.550986    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda19e41e3_b082_4b20_98dd_33cf3c50dc70.slice/crio-f0b47fc3b48079f6c15745a7d0960929ac7682a91441b678697dae320ffe8a17 WatchSource:0}: Error finding container f0b47fc3b48079f6c15745a7d0960929ac7682a91441b678697dae320ffe8a17: Status 404 returned error can't find the container with id f0b47fc3b48079f6c15745a7d0960929ac7682a91441b678697dae320ffe8a17
Sep 11 18:16:25.557328 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:25.556895    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd" event={"ID":"a19e41e3-b082-4b20-98dd-33cf3c50dc70","Type":"ContainerStarted","Data":"ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f"}
Sep 11 18:16:25.557328 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:25.556950    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd" event={"ID":"a19e41e3-b082-4b20-98dd-33cf3c50dc70","Type":"ContainerStarted","Data":"f0b47fc3b48079f6c15745a7d0960929ac7682a91441b678697dae320ffe8a17"}
Sep 11 18:16:25.582319 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:25.579309    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd" podStartSLOduration=2.579269441 podStartE2EDuration="2.579269441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:25.577562862 +0000 UTC m=+1349.347629225" watchObservedRunningTime="2026-09-11 18:16:25.579269441 +0000 UTC m=+1349.349335792"
Sep 11 18:16:34.270071 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:34.270023    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:34.270071 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:34.270079    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:34.283279 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:34.283252    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:34.409495 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:34.409453    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 18:16:34.409671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:34.409530    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 18:16:34.636087 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:34.636008    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:16:36.638998 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:36.638967    2571 generic.go:356] "Generic (PLEG): container finished" podID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerID="7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f" exitCode=0
Sep 11 18:16:36.639463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:36.639039    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" event={"ID":"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b","Type":"ContainerDied","Data":"7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f"}
Sep 11 18:16:36.639463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:36.639426    2571 scope.go:117] "RemoveContainer" containerID="7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f"
Sep 11 18:16:36.640500 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:36.640471    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:16:36.640617 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:36.640553    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:16:36.640617 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:36.640603    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:16:37.648602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:37.648539    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" event={"ID":"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b","Type":"ContainerStarted","Data":"f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3"}
Sep 11 18:16:38.322062 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:38.322027    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:38.322248 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:38.322225    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 11 18:16:38.322312 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:38.322278    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 11 18:16:39.321727 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:39.321686    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 11 18:16:39.322199 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:39.321746    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 11 18:16:40.321812 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:40.321772    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 11 18:16:40.322205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:40.321844    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 11 18:16:41.322669 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:41.322629    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 11 18:16:41.323064 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:41.322690    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 11 18:16:42.322521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:42.322488    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Sep 11 18:16:42.322695 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:42.322547    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Sep 11 18:16:43.325550 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:43.325517    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:43.325933 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:43.325854    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:43.326672 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:43.326655    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:16:45.396855 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.396823    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"]
Sep 11 18:16:45.401819 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.401801    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:45.404575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.404549    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5d35e7\""
Sep 11 18:16:45.404791 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.404771    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5d35e7-dockercfg-ps4b7\""
Sep 11 18:16:45.413598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.413577    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"]
Sep 11 18:16:45.504283 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.504232    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215a093e-eb0b-441d-b7b2-bcf9d911c846-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d\" (UID: \"215a093e-eb0b-441d-b7b2-bcf9d911c846\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:45.504502 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.504359    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8d4\" (UniqueName: \"kubernetes.io/projected/215a093e-eb0b-441d-b7b2-bcf9d911c846-kube-api-access-5p8d4\") pod \"payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d\" (UID: \"215a093e-eb0b-441d-b7b2-bcf9d911c846\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:45.605151 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.605107    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215a093e-eb0b-441d-b7b2-bcf9d911c846-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d\" (UID: \"215a093e-eb0b-441d-b7b2-bcf9d911c846\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:45.605404 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.605201    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8d4\" (UniqueName: \"kubernetes.io/projected/215a093e-eb0b-441d-b7b2-bcf9d911c846-kube-api-access-5p8d4\") pod \"payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d\" (UID: \"215a093e-eb0b-441d-b7b2-bcf9d911c846\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:45.605903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.605876    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215a093e-eb0b-441d-b7b2-bcf9d911c846-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d\" (UID: \"215a093e-eb0b-441d-b7b2-bcf9d911c846\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:45.614733 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.614699    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8d4\" (UniqueName: \"kubernetes.io/projected/215a093e-eb0b-441d-b7b2-bcf9d911c846-kube-api-access-5p8d4\") pod \"payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d\" (UID: \"215a093e-eb0b-441d-b7b2-bcf9d911c846\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:45.712798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:45.712764    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:46.017249 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:46.017212    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"]
Sep 11 18:16:46.018874 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:16:46.018842    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215a093e_eb0b_441d_b7b2_bcf9d911c846.slice/crio-f41ea3dfd28b864e6cff8664a407419558f6527f2744db84d63d9ef423252bdb WatchSource:0}: Error finding container f41ea3dfd28b864e6cff8664a407419558f6527f2744db84d63d9ef423252bdb: Status 404 returned error can't find the container with id f41ea3dfd28b864e6cff8664a407419558f6527f2744db84d63d9ef423252bdb
Sep 11 18:16:46.722192 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:46.722157    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d" event={"ID":"215a093e-eb0b-441d-b7b2-bcf9d911c846","Type":"ContainerStarted","Data":"db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e"}
Sep 11 18:16:46.722611 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:46.722211    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d" event={"ID":"215a093e-eb0b-441d-b7b2-bcf9d911c846","Type":"ContainerStarted","Data":"f41ea3dfd28b864e6cff8664a407419558f6527f2744db84d63d9ef423252bdb"}
Sep 11 18:16:46.722611 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:46.722314    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:16:46.745678 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:46.745624    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d" podStartSLOduration=1.745607146 podStartE2EDuration="1.745607146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:46.743479517 +0000 UTC m=+1370.513545866" watchObservedRunningTime="2026-09-11 18:16:46.745607146 +0000 UTC m=+1370.515673494"
Sep 11 18:16:57.056881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:57.056848    2571 scope.go:117] "RemoveContainer" containerID="f689e6d5bd479bead81a30b7c70e415eec72d2d57544e2d5c3959a3def56abf4"
Sep 11 18:16:57.731625 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:16:57.731595    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:17:15.625995 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.625960    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"]
Sep 11 18:17:15.631224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.631203    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.635441 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.635416    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"auth-test-model-a5bbf6-kserve-self-signed-certs\""
Sep 11 18:17:15.635576 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.635416    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"default-dockercfg-dpfk7\""
Sep 11 18:17:15.635576 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.635419    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"kube-root-ca.crt\""
Sep 11 18:17:15.635576 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.635463    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"openshift-service-ca.crt\""
Sep 11 18:17:15.640245 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.640221    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"]
Sep 11 18:17:15.698614 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.698584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpz82\" (UniqueName: \"kubernetes.io/projected/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-kube-api-access-wpz82\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.698800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.698640    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-model-cache\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.698800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.698722    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-home\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.698896 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.698839    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-dshm\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.698896 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.698873    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tls-certs\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.698896 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.698893    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tmp-dir\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.799742 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.799708    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpz82\" (UniqueName: \"kubernetes.io/projected/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-kube-api-access-wpz82\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.799910 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.799763    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-model-cache\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.799910 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.799791    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-home\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.799910 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.799854    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-dshm\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.799910 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.799899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tls-certs\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.800141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.799927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tmp-dir\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.800230 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.800192    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-model-cache\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.800390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.800207    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-home\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.800390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.800247    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tmp-dir\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.802069 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.802051    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-dshm\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.802426 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.802408    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tls-certs\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.809839 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.809815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpz82\" (UniqueName: \"kubernetes.io/projected/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-kube-api-access-wpz82\") pod \"auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:15.941501 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:15.941471    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:16.216794 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:17:16.216759    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ab54ffe_0ee4_446e_a4a4_d2af3f4ffbce.slice/crio-6d669cb2d98aa7079c95b7ae89c78e1e214feb0f50926e6a0f897d19cde278e7 WatchSource:0}: Error finding container 6d669cb2d98aa7079c95b7ae89c78e1e214feb0f50926e6a0f897d19cde278e7: Status 404 returned error can't find the container with id 6d669cb2d98aa7079c95b7ae89c78e1e214feb0f50926e6a0f897d19cde278e7
Sep 11 18:17:16.217019 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:16.216996    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"]
Sep 11 18:17:16.930961 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:16.930922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw" event={"ID":"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce","Type":"ContainerStarted","Data":"044145bdf15e35a551ceac9c882a57c7aafc0e7a5cea7598de2cd05994869e7f"}
Sep 11 18:17:16.931360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:16.930967    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw" event={"ID":"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce","Type":"ContainerStarted","Data":"6d669cb2d98aa7079c95b7ae89c78e1e214feb0f50926e6a0f897d19cde278e7"}
Sep 11 18:17:16.962481 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:16.962425    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw" podStartSLOduration=1.962407786 podStartE2EDuration="1.962407786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:16.951468801 +0000 UTC m=+1400.721535147" watchObservedRunningTime="2026-09-11 18:17:16.962407786 +0000 UTC m=+1400.732474134"
Sep 11 18:17:18.523003 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:18.522969    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"]
Sep 11 18:17:25.942066 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:25.942012    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:25.942066 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:25.942080    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:25.954853 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:25.954825    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:26.002121 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.002089    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:17:26.264504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.264429    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"]
Sep 11 18:17:26.268446 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.268428    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"
Sep 11 18:17:26.271927 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.271908    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:17:26.282562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.282539    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"]
Sep 11 18:17:26.406781 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.406740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82c81b-6484-4bf3-a691-c94605496848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3a652320-z69np\" (UID: \"7e82c81b-6484-4bf3-a691-c94605496848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"
Sep 11 18:17:26.508071 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.508032    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82c81b-6484-4bf3-a691-c94605496848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3a652320-z69np\" (UID: \"7e82c81b-6484-4bf3-a691-c94605496848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"
Sep 11 18:17:26.508673 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.508651    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82c81b-6484-4bf3-a691-c94605496848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3a652320-z69np\" (UID: \"7e82c81b-6484-4bf3-a691-c94605496848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"
Sep 11 18:17:26.579011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.578928    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"
Sep 11 18:17:26.864386 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:17:26.864354    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e82c81b_6484_4bf3_a691_c94605496848.slice/crio-ac8cf4119c2cea5fef235e5508c037e042eea1aaa38d4bd1d6febca09ee1146e WatchSource:0}: Error finding container ac8cf4119c2cea5fef235e5508c037e042eea1aaa38d4bd1d6febca09ee1146e: Status 404 returned error can't find the container with id ac8cf4119c2cea5fef235e5508c037e042eea1aaa38d4bd1d6febca09ee1146e
Sep 11 18:17:26.865183 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.865158    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"]
Sep 11 18:17:26.998235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:26.998200    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np" event={"ID":"7e82c81b-6484-4bf3-a691-c94605496848","Type":"ContainerStarted","Data":"ac8cf4119c2cea5fef235e5508c037e042eea1aaa38d4bd1d6febca09ee1146e"}
Sep 11 18:17:28.006169 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:28.006135    2571 generic.go:356] "Generic (PLEG): container finished" podID="7e82c81b-6484-4bf3-a691-c94605496848" containerID="d19f996f4de25fbd879bcf6a3bbcaac8875d069cc475b746fe050a993d97dcfe" exitCode=0
Sep 11 18:17:28.006569 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:28.006187    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np" event={"ID":"7e82c81b-6484-4bf3-a691-c94605496848","Type":"ContainerDied","Data":"d19f996f4de25fbd879bcf6a3bbcaac8875d069cc475b746fe050a993d97dcfe"}
Sep 11 18:17:29.170043 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:29.170015    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"
Sep 11 18:17:29.242432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:29.242401    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82c81b-6484-4bf3-a691-c94605496848-maas-api-service-ca\") pod \"7e82c81b-6484-4bf3-a691-c94605496848\" (UID: \"7e82c81b-6484-4bf3-a691-c94605496848\") "
Sep 11 18:17:29.242768 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:29.242746    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e82c81b-6484-4bf3-a691-c94605496848-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e82c81b-6484-4bf3-a691-c94605496848" (UID: "7e82c81b-6484-4bf3-a691-c94605496848"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:17:29.343535 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:29.343457    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e82c81b-6484-4bf3-a691-c94605496848-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:30.020822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:30.020792    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"
Sep 11 18:17:30.021013 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:30.020835    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np" event={"ID":"7e82c81b-6484-4bf3-a691-c94605496848","Type":"ContainerDied","Data":"ac8cf4119c2cea5fef235e5508c037e042eea1aaa38d4bd1d6febca09ee1146e"}
Sep 11 18:17:30.021013 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:30.020892    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac8cf4119c2cea5fef235e5508c037e042eea1aaa38d4bd1d6febca09ee1146e"
Sep 11 18:17:33.542809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.542769    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd" podUID="a19e41e3-b082-4b20-98dd-33cf3c50dc70" containerName="main" containerID="cri-o://ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f" gracePeriod=45
Sep 11 18:17:33.808043 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.807978    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:17:33.888454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.888417    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-dshm\") pod \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") "
Sep 11 18:17:33.888628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.888482    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-home\") pod \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") "
Sep 11 18:17:33.888628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.888542    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tmp-dir\") pod \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") "
Sep 11 18:17:33.888628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.888579    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-model-cache\") pod \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") "
Sep 11 18:17:33.888800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.888639    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tdkb\" (UniqueName: \"kubernetes.io/projected/a19e41e3-b082-4b20-98dd-33cf3c50dc70-kube-api-access-2tdkb\") pod \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") "
Sep 11 18:17:33.888800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.888721    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tls-certs\") pod \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\" (UID: \"a19e41e3-b082-4b20-98dd-33cf3c50dc70\") "
Sep 11 18:17:33.888800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.888739    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-home" (OuterVolumeSpecName: "home") pod "a19e41e3-b082-4b20-98dd-33cf3c50dc70" (UID: "a19e41e3-b082-4b20-98dd-33cf3c50dc70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:33.889417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.889097    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a19e41e3-b082-4b20-98dd-33cf3c50dc70" (UID: "a19e41e3-b082-4b20-98dd-33cf3c50dc70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:33.889912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.889887    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:33.890054 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.889914    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:33.894129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.890752    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-model-cache" (OuterVolumeSpecName: "model-cache") pod "a19e41e3-b082-4b20-98dd-33cf3c50dc70" (UID: "a19e41e3-b082-4b20-98dd-33cf3c50dc70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:33.896374 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.894375    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-dshm" (OuterVolumeSpecName: "dshm") pod "a19e41e3-b082-4b20-98dd-33cf3c50dc70" (UID: "a19e41e3-b082-4b20-98dd-33cf3c50dc70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:33.899266 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.899124    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19e41e3-b082-4b20-98dd-33cf3c50dc70-kube-api-access-2tdkb" (OuterVolumeSpecName: "kube-api-access-2tdkb") pod "a19e41e3-b082-4b20-98dd-33cf3c50dc70" (UID: "a19e41e3-b082-4b20-98dd-33cf3c50dc70"). InnerVolumeSpecName "kube-api-access-2tdkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:33.899546 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.899524    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a19e41e3-b082-4b20-98dd-33cf3c50dc70" (UID: "a19e41e3-b082-4b20-98dd-33cf3c50dc70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:17:33.991037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.990995    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:33.991037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.991029    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tdkb\" (UniqueName: \"kubernetes.io/projected/a19e41e3-b082-4b20-98dd-33cf3c50dc70-kube-api-access-2tdkb\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:33.991037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.991040    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a19e41e3-b082-4b20-98dd-33cf3c50dc70-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:33.991037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:33.991048    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a19e41e3-b082-4b20-98dd-33cf3c50dc70-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:34.050520 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.050487    2571 generic.go:356] "Generic (PLEG): container finished" podID="a19e41e3-b082-4b20-98dd-33cf3c50dc70" containerID="ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f" exitCode=0
Sep 11 18:17:34.050729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.050568    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"
Sep 11 18:17:34.050729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.050568    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd" event={"ID":"a19e41e3-b082-4b20-98dd-33cf3c50dc70","Type":"ContainerDied","Data":"ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f"}
Sep 11 18:17:34.050729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.050619    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd" event={"ID":"a19e41e3-b082-4b20-98dd-33cf3c50dc70","Type":"ContainerDied","Data":"f0b47fc3b48079f6c15745a7d0960929ac7682a91441b678697dae320ffe8a17"}
Sep 11 18:17:34.050729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.050640    2571 scope.go:117] "RemoveContainer" containerID="ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f"
Sep 11 18:17:34.061352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.061330    2571 scope.go:117] "RemoveContainer" containerID="ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f"
Sep 11 18:17:34.061674 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:17:34.061651    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f\": container with ID starting with ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f not found: ID does not exist" containerID="ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f"
Sep 11 18:17:34.061791 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.061681    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f"} err="failed to get container status \"ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f\": rpc error: code = NotFound desc = could not find container \"ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f\": container with ID starting with ac541ef8e90e96955b614da134bd846b81c0603303f8eac2d317232954a81e4f not found: ID does not exist"
Sep 11 18:17:34.081434 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.081404    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"]
Sep 11 18:17:34.086929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.086907    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-b-74f751c5-kserve-9f9f6d4fc-jhjvd"]
Sep 11 18:17:34.731940 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:34.731900    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19e41e3-b082-4b20-98dd-33cf3c50dc70" path="/var/lib/kubelet/pods/a19e41e3-b082-4b20-98dd-33cf3c50dc70/volumes"
Sep 11 18:17:42.139810 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.139773    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"]
Sep 11 18:17:42.142647 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.142595    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3a652320-29819175-bhqjl"]
Sep 11 18:17:42.219978 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.219940    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"]
Sep 11 18:17:42.220273 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.220218    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8" podUID="f73ca6b4-2f05-4e41-be7f-c1bd050d65bc" containerName="payload-pre-processing" containerID="cri-o://191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23" gracePeriod=30
Sep 11 18:17:42.478926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.478888    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"]
Sep 11 18:17:42.485679 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.485615    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3a652320-z69np"]
Sep 11 18:17:42.555808 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.555480    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:17:42.579957 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.579918    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgbs9\" (UniqueName: \"kubernetes.io/projected/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-kube-api-access-mgbs9\") pod \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\" (UID: \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\") "
Sep 11 18:17:42.580150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.580023    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-plugins-config-volume\") pod \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\" (UID: \"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc\") "
Sep 11 18:17:42.580483 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.580455    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f73ca6b4-2f05-4e41-be7f-c1bd050d65bc" (UID: "f73ca6b4-2f05-4e41-be7f-c1bd050d65bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:17:42.582892 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.582860    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-kube-api-access-mgbs9" (OuterVolumeSpecName: "kube-api-access-mgbs9") pod "f73ca6b4-2f05-4e41-be7f-c1bd050d65bc" (UID: "f73ca6b4-2f05-4e41-be7f-c1bd050d65bc"). InnerVolumeSpecName "kube-api-access-mgbs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:42.680682 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.680650    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:42.680682 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.680678    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbs9\" (UniqueName: \"kubernetes.io/projected/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc-kube-api-access-mgbs9\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:17:42.731779 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.731692    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43732a78-29e3-40bb-ac1c-b30272c559ca" path="/var/lib/kubelet/pods/43732a78-29e3-40bb-ac1c-b30272c559ca/volumes"
Sep 11 18:17:42.732114 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:42.732096    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e82c81b-6484-4bf3-a691-c94605496848" path="/var/lib/kubelet/pods/7e82c81b-6484-4bf3-a691-c94605496848/volumes"
Sep 11 18:17:43.120356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.120251    2571 generic.go:356] "Generic (PLEG): container finished" podID="f73ca6b4-2f05-4e41-be7f-c1bd050d65bc" containerID="191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23" exitCode=0
Sep 11 18:17:43.120356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.120329    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"
Sep 11 18:17:43.120567 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.120451    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8" event={"ID":"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc","Type":"ContainerDied","Data":"191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23"}
Sep 11 18:17:43.120567 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.120515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8" event={"ID":"f73ca6b4-2f05-4e41-be7f-c1bd050d65bc","Type":"ContainerDied","Data":"98cb19518df506a9ffc68a435d5dcab63c25fc9dea19c00e5fd514c8126f820c"}
Sep 11 18:17:43.120567 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.120549    2571 scope.go:117] "RemoveContainer" containerID="191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23"
Sep 11 18:17:43.131648 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.131602    2571 scope.go:117] "RemoveContainer" containerID="191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23"
Sep 11 18:17:43.131966 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:17:43.131947    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23\": container with ID starting with 191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23 not found: ID does not exist" containerID="191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23"
Sep 11 18:17:43.132024 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.131978    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23"} err="failed to get container status \"191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23\": rpc error: code = NotFound desc = could not find container \"191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23\": container with ID starting with 191eae4d34fc8eccd518c3a5bbd7d7e2dc2557710f50cc3834d364a9a9ffcc23 not found: ID does not exist"
Sep 11 18:17:43.155809 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.155773    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"]
Sep 11 18:17:43.166380 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:43.166345    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3a652320-5f96cb5766-6z7f8"]
Sep 11 18:17:44.731188 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:44.731150    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f73ca6b4-2f05-4e41-be7f-c1bd050d65bc" path="/var/lib/kubelet/pods/f73ca6b4-2f05-4e41-be7f-c1bd050d65bc/volumes"
Sep 11 18:17:50.686542 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:17:50.686467    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"]
Sep 11 18:18:05.708185 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:05.708120    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4" podUID="0802c482-a657-4fb1-ba1f-90a6038584b0" containerName="main" containerID="cri-o://0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d" gracePeriod=45
Sep 11 18:18:05.967692 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:05.967625    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:18:06.112971 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.112935    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-model-cache\") pod \"0802c482-a657-4fb1-ba1f-90a6038584b0\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") "
Sep 11 18:18:06.113175 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.112987    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqswr\" (UniqueName: \"kubernetes.io/projected/0802c482-a657-4fb1-ba1f-90a6038584b0-kube-api-access-nqswr\") pod \"0802c482-a657-4fb1-ba1f-90a6038584b0\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") "
Sep 11 18:18:06.113175 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113035    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-home\") pod \"0802c482-a657-4fb1-ba1f-90a6038584b0\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") "
Sep 11 18:18:06.113175 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113078    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0802c482-a657-4fb1-ba1f-90a6038584b0-tls-certs\") pod \"0802c482-a657-4fb1-ba1f-90a6038584b0\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") "
Sep 11 18:18:06.113175 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113121    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-dshm\") pod \"0802c482-a657-4fb1-ba1f-90a6038584b0\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") "
Sep 11 18:18:06.113175 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113135    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-tmp-dir\") pod \"0802c482-a657-4fb1-ba1f-90a6038584b0\" (UID: \"0802c482-a657-4fb1-ba1f-90a6038584b0\") "
Sep 11 18:18:06.113463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113264    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "0802c482-a657-4fb1-ba1f-90a6038584b0" (UID: "0802c482-a657-4fb1-ba1f-90a6038584b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:06.113463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113314    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-home" (OuterVolumeSpecName: "home") pod "0802c482-a657-4fb1-ba1f-90a6038584b0" (UID: "0802c482-a657-4fb1-ba1f-90a6038584b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:06.113571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113469    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0802c482-a657-4fb1-ba1f-90a6038584b0" (UID: "0802c482-a657-4fb1-ba1f-90a6038584b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:06.113571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113500    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:06.113571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.113516    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:06.115410 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.115389    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0802c482-a657-4fb1-ba1f-90a6038584b0-kube-api-access-nqswr" (OuterVolumeSpecName: "kube-api-access-nqswr") pod "0802c482-a657-4fb1-ba1f-90a6038584b0" (UID: "0802c482-a657-4fb1-ba1f-90a6038584b0"). InnerVolumeSpecName "kube-api-access-nqswr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:06.115655 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.115636    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-dshm" (OuterVolumeSpecName: "dshm") pod "0802c482-a657-4fb1-ba1f-90a6038584b0" (UID: "0802c482-a657-4fb1-ba1f-90a6038584b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:06.115699 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.115651    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0802c482-a657-4fb1-ba1f-90a6038584b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0802c482-a657-4fb1-ba1f-90a6038584b0" (UID: "0802c482-a657-4fb1-ba1f-90a6038584b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:06.214792 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.214759    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqswr\" (UniqueName: \"kubernetes.io/projected/0802c482-a657-4fb1-ba1f-90a6038584b0-kube-api-access-nqswr\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:06.214792 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.214790    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0802c482-a657-4fb1-ba1f-90a6038584b0-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:06.214792 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.214799    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:06.215033 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.214806    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0802c482-a657-4fb1-ba1f-90a6038584b0-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:06.274885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.274783    2571 generic.go:356] "Generic (PLEG): container finished" podID="0802c482-a657-4fb1-ba1f-90a6038584b0" containerID="0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d" exitCode=0
Sep 11 18:18:06.274885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.274832    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"
Sep 11 18:18:06.274885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.274874    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4" event={"ID":"0802c482-a657-4fb1-ba1f-90a6038584b0","Type":"ContainerDied","Data":"0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d"}
Sep 11 18:18:06.275206 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.274914    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4" event={"ID":"0802c482-a657-4fb1-ba1f-90a6038584b0","Type":"ContainerDied","Data":"d5131b790f2a85d06aeea9842bfa9f9201daefaa1f0186596a0ed6c725669915"}
Sep 11 18:18:06.275206 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.274932    2571 scope.go:117] "RemoveContainer" containerID="0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d"
Sep 11 18:18:06.285584 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.285567    2571 scope.go:117] "RemoveContainer" containerID="0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d"
Sep 11 18:18:06.285850 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:18:06.285831    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d\": container with ID starting with 0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d not found: ID does not exist" containerID="0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d"
Sep 11 18:18:06.285901 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.285858    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d"} err="failed to get container status \"0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d\": rpc error: code = NotFound desc = could not find container \"0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d\": container with ID starting with 0919f818d02f184962a2733660ea191666f13218f6f95a2debb484a9625efd6d not found: ID does not exist"
Sep 11 18:18:06.299966 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.299941    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"]
Sep 11 18:18:06.306415 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.306393    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3a652320/e2e-lifecycle-model-3a652320-kserve-8496d988c-vf7w4"]
Sep 11 18:18:06.731503 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:06.731462    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0802c482-a657-4fb1-ba1f-90a6038584b0" path="/var/lib/kubelet/pods/0802c482-a657-4fb1-ba1f-90a6038584b0/volumes"
Sep 11 18:18:33.919624 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.919593    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"]
Sep 11 18:18:33.920078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920025    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73ca6b4-2f05-4e41-be7f-c1bd050d65bc" containerName="payload-pre-processing"
Sep 11 18:18:33.920078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920036    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73ca6b4-2f05-4e41-be7f-c1bd050d65bc" containerName="payload-pre-processing"
Sep 11 18:18:33.920078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920052    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19e41e3-b082-4b20-98dd-33cf3c50dc70" containerName="main"
Sep 11 18:18:33.920078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920057    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19e41e3-b082-4b20-98dd-33cf3c50dc70" containerName="main"
Sep 11 18:18:33.920078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920069    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0802c482-a657-4fb1-ba1f-90a6038584b0" containerName="main"
Sep 11 18:18:33.920078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920074    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0802c482-a657-4fb1-ba1f-90a6038584b0" containerName="main"
Sep 11 18:18:33.920275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920082    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e82c81b-6484-4bf3-a691-c94605496848" containerName="revoke-keys"
Sep 11 18:18:33.920275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920088    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e82c81b-6484-4bf3-a691-c94605496848" containerName="revoke-keys"
Sep 11 18:18:33.920275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920153    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73ca6b4-2f05-4e41-be7f-c1bd050d65bc" containerName="payload-pre-processing"
Sep 11 18:18:33.920275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920163    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0802c482-a657-4fb1-ba1f-90a6038584b0" containerName="main"
Sep 11 18:18:33.920275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920171    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19e41e3-b082-4b20-98dd-33cf3c50dc70" containerName="main"
Sep 11 18:18:33.920275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.920178    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e82c81b-6484-4bf3-a691-c94605496848" containerName="revoke-keys"
Sep 11 18:18:33.922876 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.922859    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:33.927423 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.927394    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"kube-root-ca.crt\""
Sep 11 18:18:33.927423 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.927415    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"distinct-2-w4-4e0b0b-kserve-self-signed-certs\""
Sep 11 18:18:33.927586 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.927417    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"default-dockercfg-fzqfp\""
Sep 11 18:18:33.927586 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.927440    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"openshift-service-ca.crt\""
Sep 11 18:18:33.934998 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.934974    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"]
Sep 11 18:18:33.981582 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.981542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tls-certs\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:33.981754 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.981662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-home\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:33.981754 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.981700    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tmp-dir\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:33.981754 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.981727    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-model-cache\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:33.981931 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.981783    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-dshm\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:33.981931 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:33.981826    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwk2g\" (UniqueName: \"kubernetes.io/projected/11e8865e-ae35-48dc-b3b8-0b5538e16d61-kube-api-access-dwk2g\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.083635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.082733    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-home\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.083635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.082817    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tmp-dir\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.083635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.082847    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-model-cache\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.083635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.082882    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-dshm\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.083635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.082921    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwk2g\" (UniqueName: \"kubernetes.io/projected/11e8865e-ae35-48dc-b3b8-0b5538e16d61-kube-api-access-dwk2g\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.083635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.082973    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tls-certs\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.083635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.083160    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-home\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.083635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.083461    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-model-cache\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.084207 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.083798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tmp-dir\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.086883 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.086860    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-dshm\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.087053 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.087013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tls-certs\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.094548 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.094496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwk2g\" (UniqueName: \"kubernetes.io/projected/11e8865e-ae35-48dc-b3b8-0b5538e16d61-kube-api-access-dwk2g\") pod \"distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.232728 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.232647    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:34.521786 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:34.521761    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"]
Sep 11 18:18:34.522521 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:18:34.522492    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11e8865e_ae35_48dc_b3b8_0b5538e16d61.slice/crio-ee62c01a3affdac51da137371158ecc3228b1d0be4ecb0da16f3bb82ed75349b WatchSource:0}: Error finding container ee62c01a3affdac51da137371158ecc3228b1d0be4ecb0da16f3bb82ed75349b: Status 404 returned error can't find the container with id ee62c01a3affdac51da137371158ecc3228b1d0be4ecb0da16f3bb82ed75349b
Sep 11 18:18:35.480204 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:35.480170    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl" event={"ID":"11e8865e-ae35-48dc-b3b8-0b5538e16d61","Type":"ContainerStarted","Data":"21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e"}
Sep 11 18:18:35.480656 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:35.480212    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl" event={"ID":"11e8865e-ae35-48dc-b3b8-0b5538e16d61","Type":"ContainerStarted","Data":"ee62c01a3affdac51da137371158ecc3228b1d0be4ecb0da16f3bb82ed75349b"}
Sep 11 18:18:35.500640 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:35.500594    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl" podStartSLOduration=2.500578444 podStartE2EDuration="2.500578444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:35.498431032 +0000 UTC m=+1479.268497391" watchObservedRunningTime="2026-09-11 18:18:35.500578444 +0000 UTC m=+1479.270644793"
Sep 11 18:18:41.593859 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:41.593824    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"]
Sep 11 18:18:41.594376 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:41.594065    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy" containerID="cri-o://3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8" gracePeriod=30
Sep 11 18:18:44.233258 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:44.233213    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:44.233657 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:44.233493    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:44.245901 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:44.245875    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:44.550045 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:44.549961    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:18:46.859468 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:46.859404    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:18:47.007664 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007631    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r74w\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-kube-api-access-4r74w\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.007882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007709    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istiod-ca-cert\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.007882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007731    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-podinfo\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.007882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007766    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-socket\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.007882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007814    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-token\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.007882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007835    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-credential-socket\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.007882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007854    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-certs\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.007882 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007880    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-data\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.008263 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.007928    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-envoy\") pod \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\" (UID: \"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd\") "
Sep 11 18:18:47.008263 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.008142    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:47.008263 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.008195    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:47.008263 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.008237    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:47.008521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.008415    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.008521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.008437    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.008521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.008455    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.008521 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.008433    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:47.008738 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.008525    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:47.009974 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.009944    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:47.009974 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.009964    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:18:47.010111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.009986    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-kube-api-access-4r74w" (OuterVolumeSpecName: "kube-api-access-4r74w") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "kube-api-access-4r74w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:47.010162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.010145    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" (UID: "11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:47.109873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.109785    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.109873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.109815    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.109873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.109827    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.109873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.109838    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.109873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.109849    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.109873 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.109860    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r74w\" (UniqueName: \"kubernetes.io/projected/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd-kube-api-access-4r74w\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:18:47.559035 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.558998    2571 generic.go:356] "Generic (PLEG): container finished" podID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerID="3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8" exitCode=0
Sep 11 18:18:47.559231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.559053    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"
Sep 11 18:18:47.559231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.559090    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" event={"ID":"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd","Type":"ContainerDied","Data":"3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8"}
Sep 11 18:18:47.559231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.559159    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf" event={"ID":"11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd","Type":"ContainerDied","Data":"3ea7faa00d624cd09c4e3e2e4d23492199cec3f98384066eebf28a51290bcbb5"}
Sep 11 18:18:47.559231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.559189    2571 scope.go:117] "RemoveContainer" containerID="3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8"
Sep 11 18:18:47.570215 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.570200    2571 scope.go:117] "RemoveContainer" containerID="2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8"
Sep 11 18:18:47.579510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.579494    2571 scope.go:117] "RemoveContainer" containerID="3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8"
Sep 11 18:18:47.579795 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:18:47.579776    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8\": container with ID starting with 3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8 not found: ID does not exist" containerID="3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8"
Sep 11 18:18:47.579849 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.579805    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8"} err="failed to get container status \"3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8\": rpc error: code = NotFound desc = could not find container \"3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8\": container with ID starting with 3446ec8736b4e55d8e71e2a5357d2f45a71ebd3db476be9d5b8507adad9d1fd8 not found: ID does not exist"
Sep 11 18:18:47.579849 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.579821    2571 scope.go:117] "RemoveContainer" containerID="2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8"
Sep 11 18:18:47.580060 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:18:47.580042    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8\": container with ID starting with 2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8 not found: ID does not exist" containerID="2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8"
Sep 11 18:18:47.580105 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.580066    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8"} err="failed to get container status \"2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8\": rpc error: code = NotFound desc = could not find container \"2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8\": container with ID starting with 2125ce374d8cda37bafb5dbabf2d0c307e20afc9bfb05e91b64efad37616ccb8 not found: ID does not exist"
Sep 11 18:18:47.589264 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.589244    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"]
Sep 11 18:18:47.594413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:47.594394    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3a652320-openshift-default-64c9fffd96-h5fvf"]
Sep 11 18:18:48.730885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:48.730855    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" path="/var/lib/kubelet/pods/11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd/volumes"
Sep 11 18:18:57.106037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:18:57.106007    2571 scope.go:117] "RemoveContainer" containerID="85a313c685c89dd8ed046fd8a05f7a553ee57998607181d5a3e970d1374b113f"
Sep 11 18:19:17.601432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.601392    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"]
Sep 11 18:19:17.602077 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.602057    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy"
Sep 11 18:19:17.602167 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.602080    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy"
Sep 11 18:19:17.602167 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.602103    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy"
Sep 11 18:19:17.602167 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.602111    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy"
Sep 11 18:19:17.602342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.602227    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy"
Sep 11 18:19:17.602342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.602242    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="11b7a058-ba23-48ab-a0ce-8fbfbc7db0fd" containerName="istio-proxy"
Sep 11 18:19:17.607085 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.607065    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.610359 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.610338    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"embedding-w0-f8a2d4-kserve-self-signed-certs\""
Sep 11 18:19:17.616102 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.616080    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"]
Sep 11 18:19:17.711011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.710979    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-home\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.711191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.711037    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-dshm\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.711191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.711074    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-tmp-dir\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.711191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.711106    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-model-cache\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.711191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.711144    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88f8876-741b-4871-8113-69a3700a3a18-tls-certs\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.711191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.711187    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzppr\" (UniqueName: \"kubernetes.io/projected/b88f8876-741b-4871-8113-69a3700a3a18-kube-api-access-zzppr\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812133 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812091    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-model-cache\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812133 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812139    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88f8876-741b-4871-8113-69a3700a3a18-tls-certs\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812441 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812189    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzppr\" (UniqueName: \"kubernetes.io/projected/b88f8876-741b-4871-8113-69a3700a3a18-kube-api-access-zzppr\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812441 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812286    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-home\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812573 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812545    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-dshm\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812573 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-tmp-dir\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812686 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812577    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-model-cache\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812686 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812658    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-home\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.812855 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.812834    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-tmp-dir\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.814737 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.814717    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-dshm\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.814967 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.814949    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88f8876-741b-4871-8113-69a3700a3a18-tls-certs\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.836555 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.836523    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzppr\" (UniqueName: \"kubernetes.io/projected/b88f8876-741b-4871-8113-69a3700a3a18-kube-api-access-zzppr\") pod \"embedding-w0-f8a2d4-kserve-8496d55855-47wbq\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") " pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:17.917808 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:17.917774    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:18.206872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:18.206836    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"]
Sep 11 18:19:18.207996 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:19:18.207972    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb88f8876_741b_4871_8113_69a3700a3a18.slice/crio-68a209f3d38938db9717e6382dd7d24be9d39e2e62f7ce711536d96d1a4bf3c8 WatchSource:0}: Error finding container 68a209f3d38938db9717e6382dd7d24be9d39e2e62f7ce711536d96d1a4bf3c8: Status 404 returned error can't find the container with id 68a209f3d38938db9717e6382dd7d24be9d39e2e62f7ce711536d96d1a4bf3c8
Sep 11 18:19:18.766350 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:18.766310    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq" event={"ID":"b88f8876-741b-4871-8113-69a3700a3a18","Type":"ContainerStarted","Data":"da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc"}
Sep 11 18:19:18.766817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:18.766358    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq" event={"ID":"b88f8876-741b-4871-8113-69a3700a3a18","Type":"ContainerStarted","Data":"68a209f3d38938db9717e6382dd7d24be9d39e2e62f7ce711536d96d1a4bf3c8"}
Sep 11 18:19:18.789317 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:18.789235    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq" podStartSLOduration=1.7892168320000001 podStartE2EDuration="1.789216832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:18.787388552 +0000 UTC m=+1522.557454899" watchObservedRunningTime="2026-09-11 18:19:18.789216832 +0000 UTC m=+1522.559283181"
Sep 11 18:19:27.918727 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:27.918685    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:27.918727 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:27.918731    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:27.931108 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:27.931085    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:28.840051 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:28.840024    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:19:39.877441 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:39.877410    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"]
Sep 11 18:19:39.881109 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:39.881093    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:39.885157 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:39.885138    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-684bbd-openshift-default-dockercfg-jk725\""
Sep 11 18:19:39.897766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:39.897742    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"]
Sep 11 18:19:40.024700 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.024662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-token\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.024915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.024713    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-data\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.024915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.024746    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e120ce16-8fa8-45e4-bede-d46bba3095a8-istiod-ca-cert\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.024915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.024813    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-podinfo\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.024915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.024834    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-envoy\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.025081 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.024950    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8cwp\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-kube-api-access-b8cwp\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.025081 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.024988    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-credential-socket\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.025081 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.025009    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-socket\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.025205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.025082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-certs\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.125993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.125954    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-token\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.125993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.125993    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-data\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126016    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e120ce16-8fa8-45e4-bede-d46bba3095a8-istiod-ca-cert\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126043    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-podinfo\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-envoy\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126133    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8cwp\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-kube-api-access-b8cwp\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-credential-socket\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-socket\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-certs\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126423    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-data\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126730 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-credential-socket\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126730 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126721    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-socket\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126782    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-certs\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.126917 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.126893    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e120ce16-8fa8-45e4-bede-d46bba3095a8-istiod-ca-cert\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.128410 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.128366    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-envoy\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.128491 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.128475    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-podinfo\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.144339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.144311    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-token\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.144446 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.144358    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8cwp\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-kube-api-access-b8cwp\") pod \"e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") " pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.191735 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.191711    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:40.472927 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.472902    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"]
Sep 11 18:19:40.473222 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:19:40.473201    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode120ce16_8fa8_45e4_bede_d46bba3095a8.slice/crio-4490e845c13cca5727b4cd3339397f955475b11eaa65a3221e7ec5b1827aaf34 WatchSource:0}: Error finding container 4490e845c13cca5727b4cd3339397f955475b11eaa65a3221e7ec5b1827aaf34: Status 404 returned error can't find the container with id 4490e845c13cca5727b4cd3339397f955475b11eaa65a3221e7ec5b1827aaf34
Sep 11 18:19:40.475512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.475483    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:19:40.475584 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.475541    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:19:40.475584 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.475569    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:19:40.915386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.915346    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" event={"ID":"e120ce16-8fa8-45e4-bede-d46bba3095a8","Type":"ContainerStarted","Data":"c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79"}
Sep 11 18:19:40.915839 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.915404    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" event={"ID":"e120ce16-8fa8-45e4-bede-d46bba3095a8","Type":"ContainerStarted","Data":"4490e845c13cca5727b4cd3339397f955475b11eaa65a3221e7ec5b1827aaf34"}
Sep 11 18:19:40.942946 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:40.942897    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" podStartSLOduration=1.942883125 podStartE2EDuration="1.942883125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:40.941401791 +0000 UTC m=+1544.711468139" watchObservedRunningTime="2026-09-11 18:19:40.942883125 +0000 UTC m=+1544.712949474"
Sep 11 18:19:41.192886 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:41.192813    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:41.198456 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:41.198433    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:41.921810 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:41.921778    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:41.922596 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:41.922578    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:19:51.305923 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.305887    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"]
Sep 11 18:19:51.310053 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.310030    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:51.313221 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.313200    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-684bbd\""
Sep 11 18:19:51.313356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.313236    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-684bbd-dockercfg-ptshq\""
Sep 11 18:19:51.323959 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.323935    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"]
Sep 11 18:19:51.333743 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.333717    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97898420-b022-4da9-b765-d7b97df22002-plugins-config-volume\") pod \"payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp\" (UID: \"97898420-b022-4da9-b765-d7b97df22002\") " pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:51.333877 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.333809    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55tj\" (UniqueName: \"kubernetes.io/projected/97898420-b022-4da9-b765-d7b97df22002-kube-api-access-v55tj\") pod \"payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp\" (UID: \"97898420-b022-4da9-b765-d7b97df22002\") " pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:51.435381 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.435328    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97898420-b022-4da9-b765-d7b97df22002-plugins-config-volume\") pod \"payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp\" (UID: \"97898420-b022-4da9-b765-d7b97df22002\") " pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:51.435598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.435417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55tj\" (UniqueName: \"kubernetes.io/projected/97898420-b022-4da9-b765-d7b97df22002-kube-api-access-v55tj\") pod \"payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp\" (UID: \"97898420-b022-4da9-b765-d7b97df22002\") " pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:51.436134 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.436108    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97898420-b022-4da9-b765-d7b97df22002-plugins-config-volume\") pod \"payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp\" (UID: \"97898420-b022-4da9-b765-d7b97df22002\") " pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:51.446328 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.446267    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55tj\" (UniqueName: \"kubernetes.io/projected/97898420-b022-4da9-b765-d7b97df22002-kube-api-access-v55tj\") pod \"payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp\" (UID: \"97898420-b022-4da9-b765-d7b97df22002\") " pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:51.621844 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.621747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:51.898855 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.898823    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"]
Sep 11 18:19:51.899518 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:19:51.899492    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97898420_b022_4da9_b765_d7b97df22002.slice/crio-9547859c1fb08875e74f58936258460bf219cdcde3c6026c6835f8923d0b859f WatchSource:0}: Error finding container 9547859c1fb08875e74f58936258460bf219cdcde3c6026c6835f8923d0b859f: Status 404 returned error can't find the container with id 9547859c1fb08875e74f58936258460bf219cdcde3c6026c6835f8923d0b859f
Sep 11 18:19:51.992682 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.992644    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp" event={"ID":"97898420-b022-4da9-b765-d7b97df22002","Type":"ContainerStarted","Data":"cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee"}
Sep 11 18:19:51.992820 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.992705    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp" event={"ID":"97898420-b022-4da9-b765-d7b97df22002","Type":"ContainerStarted","Data":"9547859c1fb08875e74f58936258460bf219cdcde3c6026c6835f8923d0b859f"}
Sep 11 18:19:51.992893 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:51.992867    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:19:52.011938 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:52.011884    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp" podStartSLOduration=1.011868566 podStartE2EDuration="1.011868566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:52.008898378 +0000 UTC m=+1555.778964762" watchObservedRunningTime="2026-09-11 18:19:52.011868566 +0000 UTC m=+1555.781934976"
Sep 11 18:19:57.142994 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:19:57.142958    2571 scope.go:117] "RemoveContainer" containerID="f1e79c7e1cae7f5d18740ea729d1483c0f1952a9bbd39e3fd94179992c994ddf"
Sep 11 18:20:03.001237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:03.001205    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:20:10.119869 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:10.119832    2571 generic.go:356] "Generic (PLEG): container finished" podID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerID="448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98" exitCode=137
Sep 11 18:20:10.120293 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:10.119907    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc" event={"ID":"64419844-ea38-4dc6-b2bd-6b5e66eaa060","Type":"ContainerDied","Data":"448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98"}
Sep 11 18:20:10.120293 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:10.120254    2571 scope.go:117] "RemoveContainer" containerID="448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98"
Sep 11 18:20:11.128080 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:11.128051    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc" event={"ID":"64419844-ea38-4dc6-b2bd-6b5e66eaa060","Type":"ContainerStarted","Data":"12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea"}
Sep 11 18:20:11.128469 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:11.128249    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:20:22.137916 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:22.137886    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:20:45.665561 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:45.665519    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"]
Sep 11 18:20:45.670237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:45.670214    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"
Sep 11 18:20:45.673651 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:45.673628    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:20:45.685308 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:45.685272    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"]
Sep 11 18:20:45.787089 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:45.787056    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe406a1-4f81-46a1-b78a-116d72bef30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2\" (UID: \"fbe406a1-4f81-46a1-b78a-116d72bef30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"
Sep 11 18:20:45.888740 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:45.888704    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe406a1-4f81-46a1-b78a-116d72bef30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2\" (UID: \"fbe406a1-4f81-46a1-b78a-116d72bef30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"
Sep 11 18:20:45.889407 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:45.889388    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe406a1-4f81-46a1-b78a-116d72bef30e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2\" (UID: \"fbe406a1-4f81-46a1-b78a-116d72bef30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"
Sep 11 18:20:45.986053 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:45.985961    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"
Sep 11 18:20:46.274865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:46.269950    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"]
Sep 11 18:20:46.369991 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:46.369950    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2" event={"ID":"fbe406a1-4f81-46a1-b78a-116d72bef30e","Type":"ContainerStarted","Data":"8c434c4434c3a61efed051434cc52bdf4a2d1ab1ea869ee243140def7faeb16d"}
Sep 11 18:20:47.377875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:47.377780    2571 generic.go:356] "Generic (PLEG): container finished" podID="fbe406a1-4f81-46a1-b78a-116d72bef30e" containerID="2b2e1f96d631c0231c04862ab95febb085aca830df51bf27b4e6cf20cb8c0942" exitCode=0
Sep 11 18:20:47.377875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:47.377846    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2" event={"ID":"fbe406a1-4f81-46a1-b78a-116d72bef30e","Type":"ContainerDied","Data":"2b2e1f96d631c0231c04862ab95febb085aca830df51bf27b4e6cf20cb8c0942"}
Sep 11 18:20:48.520954 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:48.520931    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"
Sep 11 18:20:48.617996 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:48.617968    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe406a1-4f81-46a1-b78a-116d72bef30e-maas-api-service-ca\") pod \"fbe406a1-4f81-46a1-b78a-116d72bef30e\" (UID: \"fbe406a1-4f81-46a1-b78a-116d72bef30e\") "
Sep 11 18:20:48.618741 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:48.618669    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe406a1-4f81-46a1-b78a-116d72bef30e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbe406a1-4f81-46a1-b78a-116d72bef30e" (UID: "fbe406a1-4f81-46a1-b78a-116d72bef30e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:48.719037 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:48.719000    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbe406a1-4f81-46a1-b78a-116d72bef30e-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:20:49.395272 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:49.395240    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"
Sep 11 18:20:49.395272 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:49.395255    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2" event={"ID":"fbe406a1-4f81-46a1-b78a-116d72bef30e","Type":"ContainerDied","Data":"8c434c4434c3a61efed051434cc52bdf4a2d1ab1ea869ee243140def7faeb16d"}
Sep 11 18:20:49.395504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:49.395318    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c434c4434c3a61efed051434cc52bdf4a2d1ab1ea869ee243140def7faeb16d"
Sep 11 18:20:50.201899 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.201865    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"]
Sep 11 18:20:50.202613 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.202587    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbe406a1-4f81-46a1-b78a-116d72bef30e" containerName="revoke-keys"
Sep 11 18:20:50.202613 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.202613    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbe406a1-4f81-46a1-b78a-116d72bef30e" containerName="revoke-keys"
Sep 11 18:20:50.202771 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.202715    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbe406a1-4f81-46a1-b78a-116d72bef30e" containerName="revoke-keys"
Sep 11 18:20:50.205959 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.205944    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"
Sep 11 18:20:50.208628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.208611    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:20:50.214960 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.214938    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"]
Sep 11 18:20:50.336437 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.336399    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ded8ce5-c2b1-4214-8b5d-138436ca8adb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw\" (UID: \"5ded8ce5-c2b1-4214-8b5d-138436ca8adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"
Sep 11 18:20:50.403554 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.403525    2571 generic.go:356] "Generic (PLEG): container finished" podID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerID="c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79" exitCode=0
Sep 11 18:20:50.403747 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.403602    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" event={"ID":"e120ce16-8fa8-45e4-bede-d46bba3095a8","Type":"ContainerDied","Data":"c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79"}
Sep 11 18:20:50.404086 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.404068    2571 scope.go:117] "RemoveContainer" containerID="c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79"
Sep 11 18:20:50.405243 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.405213    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:20:50.405353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.405328    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:20:50.405427 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.405404    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:20:50.437820 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.437792    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ded8ce5-c2b1-4214-8b5d-138436ca8adb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw\" (UID: \"5ded8ce5-c2b1-4214-8b5d-138436ca8adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"
Sep 11 18:20:50.438636 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.438614    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ded8ce5-c2b1-4214-8b5d-138436ca8adb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw\" (UID: \"5ded8ce5-c2b1-4214-8b5d-138436ca8adb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"
Sep 11 18:20:50.517896 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.517861    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"
Sep 11 18:20:50.581699 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.581239    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"]
Sep 11 18:20:50.599068 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.599026    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"]
Sep 11 18:20:50.800196 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:50.800167    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"]
Sep 11 18:20:50.800364 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:20:50.800237    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ded8ce5_c2b1_4214_8b5d_138436ca8adb.slice/crio-e3bb6517c081b5650748728abcc5e0861eae5dd0fe3096a54a80d803311dbe57 WatchSource:0}: Error finding container e3bb6517c081b5650748728abcc5e0861eae5dd0fe3096a54a80d803311dbe57: Status 404 returned error can't find the container with id e3bb6517c081b5650748728abcc5e0861eae5dd0fe3096a54a80d803311dbe57
Sep 11 18:20:51.412361 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:51.412323    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" event={"ID":"e120ce16-8fa8-45e4-bede-d46bba3095a8","Type":"ContainerStarted","Data":"b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4"}
Sep 11 18:20:51.413950 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:51.413923    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw" event={"ID":"5ded8ce5-c2b1-4214-8b5d-138436ca8adb","Type":"ContainerStarted","Data":"fd579d2ccb3b842d744dd5a6e12a381a3a4d56212c90f3776bc0e7d46812d01e"}
Sep 11 18:20:51.414071 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:51.413966    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw" event={"ID":"5ded8ce5-c2b1-4214-8b5d-138436ca8adb","Type":"ContainerStarted","Data":"e3bb6517c081b5650748728abcc5e0861eae5dd0fe3096a54a80d803311dbe57"}
Sep 11 18:20:51.458202 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:51.458159    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw" podStartSLOduration=1.458144104 podStartE2EDuration="1.458144104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:51.457775637 +0000 UTC m=+1615.227841984" watchObservedRunningTime="2026-09-11 18:20:51.458144104 +0000 UTC m=+1615.228210502"
Sep 11 18:20:52.192321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:52.192266    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:20:52.193334 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:52.193293    2571 patch_prober.go:28] interesting pod/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 11 18:20:52.193416 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:52.193359    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 11 18:20:52.422780 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:52.422743    2571 generic.go:356] "Generic (PLEG): container finished" podID="5ded8ce5-c2b1-4214-8b5d-138436ca8adb" containerID="fd579d2ccb3b842d744dd5a6e12a381a3a4d56212c90f3776bc0e7d46812d01e" exitCode=0
Sep 11 18:20:52.423178 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:52.422831    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw" event={"ID":"5ded8ce5-c2b1-4214-8b5d-138436ca8adb","Type":"ContainerDied","Data":"fd579d2ccb3b842d744dd5a6e12a381a3a4d56212c90f3776bc0e7d46812d01e"}
Sep 11 18:20:53.192590 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:53.192552    2571 patch_prober.go:28] interesting pod/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 11 18:20:53.192801 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:53.192625    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 11 18:20:53.608208 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:53.608180    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"
Sep 11 18:20:53.667552 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:53.667522    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ded8ce5-c2b1-4214-8b5d-138436ca8adb-maas-api-service-ca\") pod \"5ded8ce5-c2b1-4214-8b5d-138436ca8adb\" (UID: \"5ded8ce5-c2b1-4214-8b5d-138436ca8adb\") "
Sep 11 18:20:53.668473 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:53.668445    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ded8ce5-c2b1-4214-8b5d-138436ca8adb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ded8ce5-c2b1-4214-8b5d-138436ca8adb" (UID: "5ded8ce5-c2b1-4214-8b5d-138436ca8adb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:20:53.769272 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:53.769153    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ded8ce5-c2b1-4214-8b5d-138436ca8adb-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:20:54.192632 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:54.192594    2571 patch_prober.go:28] interesting pod/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 11 18:20:54.192803 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:54.192649    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 11 18:20:54.437622 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:54.437592    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"
Sep 11 18:20:54.437622 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:54.437615    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw" event={"ID":"5ded8ce5-c2b1-4214-8b5d-138436ca8adb","Type":"ContainerDied","Data":"e3bb6517c081b5650748728abcc5e0861eae5dd0fe3096a54a80d803311dbe57"}
Sep 11 18:20:54.437825 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:54.437656    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3bb6517c081b5650748728abcc5e0861eae5dd0fe3096a54a80d803311dbe57"
Sep 11 18:20:55.193119 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:55.193079    2571 patch_prober.go:28] interesting pod/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 11 18:20:55.193535 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:55.193148    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 11 18:20:56.192545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:56.192507    2571 patch_prober.go:28] interesting pod/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 11 18:20:56.192740 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:56.192575    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 11 18:20:56.922582 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:56.922549    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:20:57.192051 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:57.191973    2571 patch_prober.go:28] interesting pod/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Sep 11 18:20:57.192051 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:57.192033    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Sep 11 18:20:58.196494 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:58.196463    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:20:58.197353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:20:58.197330    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:21:01.399545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.399514    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"]
Sep 11 18:21:01.402385 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.402364    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f8a2d4-29819175-cc87l"]
Sep 11 18:21:01.421433 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.421406    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"]
Sep 11 18:21:01.421722 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.421671    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn" podUID="e045d9d6-fe59-48a2-9eb6-b46af76cbf00" containerName="payload-processing" containerID="cri-o://6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac" gracePeriod=30
Sep 11 18:21:01.436798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.436773    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"]
Sep 11 18:21:01.437005 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.436982    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch" podUID="7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1" containerName="payload-pre-processing" containerID="cri-o://fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82" gracePeriod=30
Sep 11 18:21:01.604286 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.604256    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"]
Sep 11 18:21:01.608055 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.607799    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f8a2d4-nn8q2"]
Sep 11 18:21:01.736877 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.736849    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:21:01.739724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.739703    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:21:01.854004 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.853969    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-plugins-config-volume\") pod \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\" (UID: \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\") "
Sep 11 18:21:01.854174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.854042    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-plugins-config-volume\") pod \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\" (UID: \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\") "
Sep 11 18:21:01.854174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.854108    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh5mn\" (UniqueName: \"kubernetes.io/projected/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-kube-api-access-sh5mn\") pod \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\" (UID: \"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1\") "
Sep 11 18:21:01.854174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.854136    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4wpq\" (UniqueName: \"kubernetes.io/projected/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-kube-api-access-d4wpq\") pod \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\" (UID: \"e045d9d6-fe59-48a2-9eb6-b46af76cbf00\") "
Sep 11 18:21:01.854412 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.854383    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e045d9d6-fe59-48a2-9eb6-b46af76cbf00" (UID: "e045d9d6-fe59-48a2-9eb6-b46af76cbf00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:01.854474 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.854414    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1" (UID: "7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:01.854532 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.854522    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:01.854590 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.854535    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:01.856199 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.856174    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-kube-api-access-sh5mn" (OuterVolumeSpecName: "kube-api-access-sh5mn") pod "7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1" (UID: "7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1"). InnerVolumeSpecName "kube-api-access-sh5mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:01.856442 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.856216    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-kube-api-access-d4wpq" (OuterVolumeSpecName: "kube-api-access-d4wpq") pod "e045d9d6-fe59-48a2-9eb6-b46af76cbf00" (UID: "e045d9d6-fe59-48a2-9eb6-b46af76cbf00"). InnerVolumeSpecName "kube-api-access-d4wpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:01.956028 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.955991    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh5mn\" (UniqueName: \"kubernetes.io/projected/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1-kube-api-access-sh5mn\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:01.956028 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:01.956024    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4wpq\" (UniqueName: \"kubernetes.io/projected/e045d9d6-fe59-48a2-9eb6-b46af76cbf00-kube-api-access-d4wpq\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:02.500005 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.499974    2571 generic.go:356] "Generic (PLEG): container finished" podID="7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1" containerID="fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82" exitCode=0
Sep 11 18:21:02.500417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.500022    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"
Sep 11 18:21:02.500417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.500054    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch" event={"ID":"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1","Type":"ContainerDied","Data":"fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82"}
Sep 11 18:21:02.500417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.500111    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch" event={"ID":"7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1","Type":"ContainerDied","Data":"6306381d93011ef05ebe5b49882a784aed408bace0dd75729150958e0e14becf"}
Sep 11 18:21:02.500417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.500143    2571 scope.go:117] "RemoveContainer" containerID="fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82"
Sep 11 18:21:02.501658 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.501583    2571 generic.go:356] "Generic (PLEG): container finished" podID="e045d9d6-fe59-48a2-9eb6-b46af76cbf00" containerID="6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac" exitCode=0
Sep 11 18:21:02.501658 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.501622    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"
Sep 11 18:21:02.501658 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.501640    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn" event={"ID":"e045d9d6-fe59-48a2-9eb6-b46af76cbf00","Type":"ContainerDied","Data":"6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac"}
Sep 11 18:21:02.501852 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.501679    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn" event={"ID":"e045d9d6-fe59-48a2-9eb6-b46af76cbf00","Type":"ContainerDied","Data":"929b37ac6a6fe05dd0e7b61db739126bb05c8376fea87bad4e0875a43a20937c"}
Sep 11 18:21:02.511318 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.511245    2571 scope.go:117] "RemoveContainer" containerID="fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82"
Sep 11 18:21:02.511570 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:02.511549    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82\": container with ID starting with fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82 not found: ID does not exist" containerID="fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82"
Sep 11 18:21:02.511658 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.511581    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82"} err="failed to get container status \"fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82\": rpc error: code = NotFound desc = could not find container \"fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82\": container with ID starting with fe91120dc70a1bb5e8c9628a0bb94b273a99b6c154667f672f307412979e7b82 not found: ID does not exist"
Sep 11 18:21:02.511658 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.511605    2571 scope.go:117] "RemoveContainer" containerID="6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac"
Sep 11 18:21:02.520946 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.520930    2571 scope.go:117] "RemoveContainer" containerID="6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac"
Sep 11 18:21:02.521276 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:02.521234    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac\": container with ID starting with 6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac not found: ID does not exist" containerID="6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac"
Sep 11 18:21:02.521465 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.521285    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac"} err="failed to get container status \"6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac\": rpc error: code = NotFound desc = could not find container \"6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac\": container with ID starting with 6bc4b6a67e13a35690e681cd60cd5fbe15d11205b914b2334c0e9bf8864122ac not found: ID does not exist"
Sep 11 18:21:02.533895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.533873    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"]
Sep 11 18:21:02.538102 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.538080    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-f8a2d4-774c48fc94-lb8ch"]
Sep 11 18:21:02.552991 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.552970    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"]
Sep 11 18:21:02.556679 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.556657    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f8a2d4-84db594f7c-chzfn"]
Sep 11 18:21:02.731360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.731329    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b6f835f-cb9d-4356-a4bc-7b3c03099802" path="/var/lib/kubelet/pods/7b6f835f-cb9d-4356-a4bc-7b3c03099802/volumes"
Sep 11 18:21:02.731711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.731698    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1" path="/var/lib/kubelet/pods/7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1/volumes"
Sep 11 18:21:02.732058 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.732047    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e045d9d6-fe59-48a2-9eb6-b46af76cbf00" path="/var/lib/kubelet/pods/e045d9d6-fe59-48a2-9eb6-b46af76cbf00/volumes"
Sep 11 18:21:02.732396 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:02.732384    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbe406a1-4f81-46a1-b78a-116d72bef30e" path="/var/lib/kubelet/pods/fbe406a1-4f81-46a1-b78a-116d72bef30e/volumes"
Sep 11 18:21:05.614659 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:05.614623    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r" podUID="b1a85d31-13ef-4db7-9722-63d1ebee6e57" containerName="main" containerID="cri-o://bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45" gracePeriod=45
Sep 11 18:21:05.634716 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:05.634683    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq" podUID="b88f8876-741b-4871-8113-69a3700a3a18" containerName="main" containerID="cri-o://da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc" gracePeriod=45
Sep 11 18:21:05.880385 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:05.880316    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:21:05.923649 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:05.923620    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:21:05.999994 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:05.999887    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tls-certs\") pod \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") "
Sep 11 18:21:05.999994 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:05.999968    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-dshm\") pod \"b88f8876-741b-4871-8113-69a3700a3a18\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") "
Sep 11 18:21:05.999994 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:05.999996    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-tmp-dir\") pod \"b88f8876-741b-4871-8113-69a3700a3a18\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") "
Sep 11 18:21:06.000318 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000252    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b88f8876-741b-4871-8113-69a3700a3a18" (UID: "b88f8876-741b-4871-8113-69a3700a3a18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.000516 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000489    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgbrz\" (UniqueName: \"kubernetes.io/projected/b1a85d31-13ef-4db7-9722-63d1ebee6e57-kube-api-access-sgbrz\") pod \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") "
Sep 11 18:21:06.000653 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000549    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzppr\" (UniqueName: \"kubernetes.io/projected/b88f8876-741b-4871-8113-69a3700a3a18-kube-api-access-zzppr\") pod \"b88f8876-741b-4871-8113-69a3700a3a18\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") "
Sep 11 18:21:06.000653 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000598    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tmp-dir\") pod \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") "
Sep 11 18:21:06.000653 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000631    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-dshm\") pod \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") "
Sep 11 18:21:06.000846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000656    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-home\") pod \"b88f8876-741b-4871-8113-69a3700a3a18\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") "
Sep 11 18:21:06.000846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000723    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-model-cache\") pod \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") "
Sep 11 18:21:06.000846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000750    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-home\") pod \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\" (UID: \"b1a85d31-13ef-4db7-9722-63d1ebee6e57\") "
Sep 11 18:21:06.000846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000775    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88f8876-741b-4871-8113-69a3700a3a18-tls-certs\") pod \"b88f8876-741b-4871-8113-69a3700a3a18\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") "
Sep 11 18:21:06.000846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.000841    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-model-cache\") pod \"b88f8876-741b-4871-8113-69a3700a3a18\" (UID: \"b88f8876-741b-4871-8113-69a3700a3a18\") "
Sep 11 18:21:06.002059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.001229    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.002059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.001405    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1a85d31-13ef-4db7-9722-63d1ebee6e57" (UID: "b1a85d31-13ef-4db7-9722-63d1ebee6e57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.002059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.001473    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-model-cache" (OuterVolumeSpecName: "model-cache") pod "b88f8876-741b-4871-8113-69a3700a3a18" (UID: "b88f8876-741b-4871-8113-69a3700a3a18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.002059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.001619    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-home" (OuterVolumeSpecName: "home") pod "b88f8876-741b-4871-8113-69a3700a3a18" (UID: "b88f8876-741b-4871-8113-69a3700a3a18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.002059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.001675    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-home" (OuterVolumeSpecName: "home") pod "b1a85d31-13ef-4db7-9722-63d1ebee6e57" (UID: "b1a85d31-13ef-4db7-9722-63d1ebee6e57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.002059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.001812    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1a85d31-13ef-4db7-9722-63d1ebee6e57" (UID: "b1a85d31-13ef-4db7-9722-63d1ebee6e57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.002916 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.002881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-dshm" (OuterVolumeSpecName: "dshm") pod "b88f8876-741b-4871-8113-69a3700a3a18" (UID: "b88f8876-741b-4871-8113-69a3700a3a18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.003367 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.003339    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1a85d31-13ef-4db7-9722-63d1ebee6e57" (UID: "b1a85d31-13ef-4db7-9722-63d1ebee6e57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:06.004729 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.004707    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88f8876-741b-4871-8113-69a3700a3a18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b88f8876-741b-4871-8113-69a3700a3a18" (UID: "b88f8876-741b-4871-8113-69a3700a3a18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:06.004930 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.004894    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a85d31-13ef-4db7-9722-63d1ebee6e57-kube-api-access-sgbrz" (OuterVolumeSpecName: "kube-api-access-sgbrz") pod "b1a85d31-13ef-4db7-9722-63d1ebee6e57" (UID: "b1a85d31-13ef-4db7-9722-63d1ebee6e57"). InnerVolumeSpecName "kube-api-access-sgbrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:06.005360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.005335    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88f8876-741b-4871-8113-69a3700a3a18-kube-api-access-zzppr" (OuterVolumeSpecName: "kube-api-access-zzppr") pod "b88f8876-741b-4871-8113-69a3700a3a18" (UID: "b88f8876-741b-4871-8113-69a3700a3a18"). InnerVolumeSpecName "kube-api-access-zzppr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:06.006152 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.006129    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-dshm" (OuterVolumeSpecName: "dshm") pod "b1a85d31-13ef-4db7-9722-63d1ebee6e57" (UID: "b1a85d31-13ef-4db7-9722-63d1ebee6e57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.102581 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102546    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102581 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102575    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbrz\" (UniqueName: \"kubernetes.io/projected/b1a85d31-13ef-4db7-9722-63d1ebee6e57-kube-api-access-sgbrz\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102581 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102587    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzppr\" (UniqueName: \"kubernetes.io/projected/b88f8876-741b-4871-8113-69a3700a3a18-kube-api-access-zzppr\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102597    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102605    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102614    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102622    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102629    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a85d31-13ef-4db7-9722-63d1ebee6e57-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102636    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b88f8876-741b-4871-8113-69a3700a3a18-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102644    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b88f8876-741b-4871-8113-69a3700a3a18-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.102822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.102651    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a85d31-13ef-4db7-9722-63d1ebee6e57-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.537877 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.537840    2571 generic.go:356] "Generic (PLEG): container finished" podID="b1a85d31-13ef-4db7-9722-63d1ebee6e57" containerID="bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45" exitCode=0
Sep 11 18:21:06.538075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.537891    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"
Sep 11 18:21:06.538075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.537928    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r" event={"ID":"b1a85d31-13ef-4db7-9722-63d1ebee6e57","Type":"ContainerDied","Data":"bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45"}
Sep 11 18:21:06.538075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.537969    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r" event={"ID":"b1a85d31-13ef-4db7-9722-63d1ebee6e57","Type":"ContainerDied","Data":"22dd269d82d643a4c1215f7f3f7fc17165487f715bd43c31fd1680e354c39fa6"}
Sep 11 18:21:06.538075 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.537986    2571 scope.go:117] "RemoveContainer" containerID="bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45"
Sep 11 18:21:06.539458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.539435    2571 generic.go:356] "Generic (PLEG): container finished" podID="b88f8876-741b-4871-8113-69a3700a3a18" containerID="da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc" exitCode=0
Sep 11 18:21:06.539565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.539481    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"
Sep 11 18:21:06.539565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.539522    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq" event={"ID":"b88f8876-741b-4871-8113-69a3700a3a18","Type":"ContainerDied","Data":"da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc"}
Sep 11 18:21:06.539565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.539558    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq" event={"ID":"b88f8876-741b-4871-8113-69a3700a3a18","Type":"ContainerDied","Data":"68a209f3d38938db9717e6382dd7d24be9d39e2e62f7ce711536d96d1a4bf3c8"}
Sep 11 18:21:06.549456 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.549438    2571 scope.go:117] "RemoveContainer" containerID="bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45"
Sep 11 18:21:06.549843 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:06.549817    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45\": container with ID starting with bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45 not found: ID does not exist" containerID="bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45"
Sep 11 18:21:06.550023 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.549852    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45"} err="failed to get container status \"bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45\": rpc error: code = NotFound desc = could not find container \"bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45\": container with ID starting with bd6279cdb5cece587db5675c68eab6af39d23f0d886321182060ad010e13ac45 not found: ID does not exist"
Sep 11 18:21:06.550023 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.549876    2571 scope.go:117] "RemoveContainer" containerID="da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc"
Sep 11 18:21:06.559862 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.559833    2571 scope.go:117] "RemoveContainer" containerID="da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc"
Sep 11 18:21:06.560120 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:06.560104    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc\": container with ID starting with da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc not found: ID does not exist" containerID="da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc"
Sep 11 18:21:06.560180 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.560125    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc"} err="failed to get container status \"da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc\": rpc error: code = NotFound desc = could not find container \"da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc\": container with ID starting with da3e23f43d579b91f02b571e95660ec2b0972d1db239d18a2340c2b55b26e8fc not found: ID does not exist"
Sep 11 18:21:06.566380 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.566355    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"]
Sep 11 18:21:06.570183 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.570162    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/embedding-w0-f8a2d4-kserve-8496d55855-47wbq"]
Sep 11 18:21:06.582719 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.582700    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"]
Sep 11 18:21:06.587070 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.587053    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-2-w0-f8a2d4-kserve-7f4dc6bf7-rwb2r"]
Sep 11 18:21:06.732800 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.732768    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a85d31-13ef-4db7-9722-63d1ebee6e57" path="/var/lib/kubelet/pods/b1a85d31-13ef-4db7-9722-63d1ebee6e57/volumes"
Sep 11 18:21:06.733201 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:06.733187    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b88f8876-741b-4871-8113-69a3700a3a18" path="/var/lib/kubelet/pods/b88f8876-741b-4871-8113-69a3700a3a18/volumes"
Sep 11 18:21:07.832751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:07.832716    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"]
Sep 11 18:21:07.833339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:07.832980    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp" podUID="97898420-b022-4da9-b765-d7b97df22002" containerName="payload-processing" containerID="cri-o://cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee" gracePeriod=30
Sep 11 18:21:08.018577 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.018537    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"]
Sep 11 18:21:08.023237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.023210    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-684bbd-zfmkw"]
Sep 11 18:21:08.098661 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.098604    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:21:08.222186 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.222152    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97898420-b022-4da9-b765-d7b97df22002-plugins-config-volume\") pod \"97898420-b022-4da9-b765-d7b97df22002\" (UID: \"97898420-b022-4da9-b765-d7b97df22002\") "
Sep 11 18:21:08.222186 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.222192    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v55tj\" (UniqueName: \"kubernetes.io/projected/97898420-b022-4da9-b765-d7b97df22002-kube-api-access-v55tj\") pod \"97898420-b022-4da9-b765-d7b97df22002\" (UID: \"97898420-b022-4da9-b765-d7b97df22002\") "
Sep 11 18:21:08.222598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.222573    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97898420-b022-4da9-b765-d7b97df22002-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97898420-b022-4da9-b765-d7b97df22002" (UID: "97898420-b022-4da9-b765-d7b97df22002"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:08.222662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.222646    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97898420-b022-4da9-b765-d7b97df22002-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:08.224326 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.224285    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97898420-b022-4da9-b765-d7b97df22002-kube-api-access-v55tj" (OuterVolumeSpecName: "kube-api-access-v55tj") pod "97898420-b022-4da9-b765-d7b97df22002" (UID: "97898420-b022-4da9-b765-d7b97df22002"). InnerVolumeSpecName "kube-api-access-v55tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:08.323311 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.323269    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55tj\" (UniqueName: \"kubernetes.io/projected/97898420-b022-4da9-b765-d7b97df22002-kube-api-access-v55tj\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:08.558981 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.558945    2571 generic.go:356] "Generic (PLEG): container finished" podID="97898420-b022-4da9-b765-d7b97df22002" containerID="cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee" exitCode=0
Sep 11 18:21:08.559150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.558997    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"
Sep 11 18:21:08.559150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.559023    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp" event={"ID":"97898420-b022-4da9-b765-d7b97df22002","Type":"ContainerDied","Data":"cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee"}
Sep 11 18:21:08.559150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.559079    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp" event={"ID":"97898420-b022-4da9-b765-d7b97df22002","Type":"ContainerDied","Data":"9547859c1fb08875e74f58936258460bf219cdcde3c6026c6835f8923d0b859f"}
Sep 11 18:21:08.559150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.559105    2571 scope.go:117] "RemoveContainer" containerID="cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee"
Sep 11 18:21:08.568931 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.568913    2571 scope.go:117] "RemoveContainer" containerID="cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee"
Sep 11 18:21:08.569195 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:08.569174    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee\": container with ID starting with cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee not found: ID does not exist" containerID="cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee"
Sep 11 18:21:08.569265 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.569207    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee"} err="failed to get container status \"cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee\": rpc error: code = NotFound desc = could not find container \"cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee\": container with ID starting with cac4f4807283eab94d56b4a099e1bf70710beb226d65721a8341ea498fd1fbee not found: ID does not exist"
Sep 11 18:21:08.584141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.584114    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"]
Sep 11 18:21:08.586759 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.586737    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-684bbd-5b97b8d6f9-k8dtp"]
Sep 11 18:21:08.732175 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.732142    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ded8ce5-c2b1-4214-8b5d-138436ca8adb" path="/var/lib/kubelet/pods/5ded8ce5-c2b1-4214-8b5d-138436ca8adb/volumes"
Sep 11 18:21:08.732553 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:08.732540    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97898420-b022-4da9-b765-d7b97df22002" path="/var/lib/kubelet/pods/97898420-b022-4da9-b765-d7b97df22002/volumes"
Sep 11 18:21:15.913395 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:15.913359    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"]
Sep 11 18:21:15.913870 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:15.913560    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy" containerID="cri-o://b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4" gracePeriod=30
Sep 11 18:21:16.243763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.243683    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"]
Sep 11 18:21:16.244111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244099    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88f8876-741b-4871-8113-69a3700a3a18" containerName="main"
Sep 11 18:21:16.244158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244112    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88f8876-741b-4871-8113-69a3700a3a18" containerName="main"
Sep 11 18:21:16.244158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244121    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1" containerName="payload-pre-processing"
Sep 11 18:21:16.244158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244127    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1" containerName="payload-pre-processing"
Sep 11 18:21:16.244158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244134    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97898420-b022-4da9-b765-d7b97df22002" containerName="payload-processing"
Sep 11 18:21:16.244158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244141    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="97898420-b022-4da9-b765-d7b97df22002" containerName="payload-processing"
Sep 11 18:21:16.244158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244151    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e045d9d6-fe59-48a2-9eb6-b46af76cbf00" containerName="payload-processing"
Sep 11 18:21:16.244158 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244156    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e045d9d6-fe59-48a2-9eb6-b46af76cbf00" containerName="payload-processing"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244171    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a85d31-13ef-4db7-9722-63d1ebee6e57" containerName="main"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244177    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a85d31-13ef-4db7-9722-63d1ebee6e57" containerName="main"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244184    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ded8ce5-c2b1-4214-8b5d-138436ca8adb" containerName="revoke-keys"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244190    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ded8ce5-c2b1-4214-8b5d-138436ca8adb" containerName="revoke-keys"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244252    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88f8876-741b-4871-8113-69a3700a3a18" containerName="main"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244261    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a85d31-13ef-4db7-9722-63d1ebee6e57" containerName="main"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244267    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e045d9d6-fe59-48a2-9eb6-b46af76cbf00" containerName="payload-processing"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244272    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ded8ce5-c2b1-4214-8b5d-138436ca8adb" containerName="revoke-keys"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244279    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="97898420-b022-4da9-b765-d7b97df22002" containerName="payload-processing"
Sep 11 18:21:16.244424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.244287    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2f9d5d-1a9b-4c79-8208-77f0a0fa4bc1" containerName="payload-pre-processing"
Sep 11 18:21:16.248916 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.248897    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"
Sep 11 18:21:16.251938 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.251901    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:21:16.256166 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.256144    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"]
Sep 11 18:21:16.398913 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.398877    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0503417-229c-463e-9fa3-292d5316131c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v\" (UID: \"c0503417-229c-463e-9fa3-292d5316131c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"
Sep 11 18:21:16.499712 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.499619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0503417-229c-463e-9fa3-292d5316131c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v\" (UID: \"c0503417-229c-463e-9fa3-292d5316131c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"
Sep 11 18:21:16.500233 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.500212    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0503417-229c-463e-9fa3-292d5316131c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v\" (UID: \"c0503417-229c-463e-9fa3-292d5316131c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"
Sep 11 18:21:16.561905 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.561871    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"
Sep 11 18:21:16.843862 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:16.843837    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"]
Sep 11 18:21:16.845807 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:21:16.845781    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0503417_229c_463e_9fa3_292d5316131c.slice/crio-eb5fb25fdc1b719b8ad71b9a40e26db94bdaac0ee008ceaec403022f016bf5dd WatchSource:0}: Error finding container eb5fb25fdc1b719b8ad71b9a40e26db94bdaac0ee008ceaec403022f016bf5dd: Status 404 returned error can't find the container with id eb5fb25fdc1b719b8ad71b9a40e26db94bdaac0ee008ceaec403022f016bf5dd
Sep 11 18:21:17.624446 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:17.624414    2571 generic.go:356] "Generic (PLEG): container finished" podID="c0503417-229c-463e-9fa3-292d5316131c" containerID="c6a38a37b71384f94e9947cd9ec77b5fbd25529c7035d7b382343bef1ba1975d" exitCode=0
Sep 11 18:21:17.624853 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:17.624499    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v" event={"ID":"c0503417-229c-463e-9fa3-292d5316131c","Type":"ContainerDied","Data":"c6a38a37b71384f94e9947cd9ec77b5fbd25529c7035d7b382343bef1ba1975d"}
Sep 11 18:21:17.624853 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:17.624565    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v" event={"ID":"c0503417-229c-463e-9fa3-292d5316131c","Type":"ContainerStarted","Data":"eb5fb25fdc1b719b8ad71b9a40e26db94bdaac0ee008ceaec403022f016bf5dd"}
Sep 11 18:21:18.773605 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:18.773581    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"
Sep 11 18:21:18.925652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:18.925623    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0503417-229c-463e-9fa3-292d5316131c-maas-api-service-ca\") pod \"c0503417-229c-463e-9fa3-292d5316131c\" (UID: \"c0503417-229c-463e-9fa3-292d5316131c\") "
Sep 11 18:21:18.925942 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:18.925920    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0503417-229c-463e-9fa3-292d5316131c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0503417-229c-463e-9fa3-292d5316131c" (UID: "c0503417-229c-463e-9fa3-292d5316131c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:19.026338 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:19.026282    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0503417-229c-463e-9fa3-292d5316131c-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:19.638435 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:19.638406    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"
Sep 11 18:21:19.638634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:19.638450    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v" event={"ID":"c0503417-229c-463e-9fa3-292d5316131c","Type":"ContainerDied","Data":"eb5fb25fdc1b719b8ad71b9a40e26db94bdaac0ee008ceaec403022f016bf5dd"}
Sep 11 18:21:19.638634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:19.638502    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5fb25fdc1b719b8ad71b9a40e26db94bdaac0ee008ceaec403022f016bf5dd"
Sep 11 18:21:21.174530 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.174471    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:21:21.249033 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249001    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-data\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249053    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8cwp\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-kube-api-access-b8cwp\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249103    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-envoy\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249129    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-certs\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249150    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-socket\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249200    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-token\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249492 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249322    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:21.249492 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249398    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-credential-socket\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249492 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249445    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e120ce16-8fa8-45e4-bede-d46bba3095a8-istiod-ca-cert\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249657 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249517    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-podinfo\") pod \"e120ce16-8fa8-45e4-bede-d46bba3095a8\" (UID: \"e120ce16-8fa8-45e4-bede-d46bba3095a8\") "
Sep 11 18:21:21.249657 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249620    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:21.249657 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249635    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:21.249812 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249753    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:21.249948 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249929    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.250014 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249956    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.250014 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249970    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.250014 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249984    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.250014 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.249966    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e120ce16-8fa8-45e4-bede-d46bba3095a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:21.251766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.251728    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:21.251985 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.251962    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-kube-api-access-b8cwp" (OuterVolumeSpecName: "kube-api-access-b8cwp") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "kube-api-access-b8cwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:21.252149 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.252124    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:21.252315 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.252277    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e120ce16-8fa8-45e4-bede-d46bba3095a8" (UID: "e120ce16-8fa8-45e4-bede-d46bba3095a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:21:21.350608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.350563    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8cwp\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-kube-api-access-b8cwp\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.350608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.350601    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.350608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.350615    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.350910 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.350628    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e120ce16-8fa8-45e4-bede-d46bba3095a8-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.350910 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.350643    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e120ce16-8fa8-45e4-bede-d46bba3095a8-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:21.654108 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.654011    2571 generic.go:356] "Generic (PLEG): container finished" podID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerID="b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4" exitCode=0
Sep 11 18:21:21.654108 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.654055    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" event={"ID":"e120ce16-8fa8-45e4-bede-d46bba3095a8","Type":"ContainerDied","Data":"b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4"}
Sep 11 18:21:21.654374 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.654069    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"
Sep 11 18:21:21.654374 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.654159    2571 scope.go:117] "RemoveContainer" containerID="b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4"
Sep 11 18:21:21.654374 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.654145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6" event={"ID":"e120ce16-8fa8-45e4-bede-d46bba3095a8","Type":"ContainerDied","Data":"4490e845c13cca5727b4cd3339397f955475b11eaa65a3221e7ec5b1827aaf34"}
Sep 11 18:21:21.664828 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.664811    2571 scope.go:117] "RemoveContainer" containerID="c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79"
Sep 11 18:21:21.673622 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.673603    2571 scope.go:117] "RemoveContainer" containerID="b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4"
Sep 11 18:21:21.673883 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:21.673862    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4\": container with ID starting with b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4 not found: ID does not exist" containerID="b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4"
Sep 11 18:21:21.673963 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.673893    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4"} err="failed to get container status \"b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4\": rpc error: code = NotFound desc = could not find container \"b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4\": container with ID starting with b370ba91b6365ed45e47304a770a46692ad5127b529423c201d42fdc827578b4 not found: ID does not exist"
Sep 11 18:21:21.673963 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.673918    2571 scope.go:117] "RemoveContainer" containerID="c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79"
Sep 11 18:21:21.674176 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:21.674157    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79\": container with ID starting with c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79 not found: ID does not exist" containerID="c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79"
Sep 11 18:21:21.674217 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.674183    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79"} err="failed to get container status \"c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79\": rpc error: code = NotFound desc = could not find container \"c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79\": container with ID starting with c4bb0b9c8cd35d5b1d4b6e6afc8998365a52dbc4300a12a9f90cdf42b1d3cc79 not found: ID does not exist"
Sep 11 18:21:21.678290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.678267    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"]
Sep 11 18:21:21.684289 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:21.684270    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-684bbd-openshift-default-6d8c4fb5d8-4lwf6"]
Sep 11 18:21:22.731058 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:22.731027    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" path="/var/lib/kubelet/pods/e120ce16-8fa8-45e4-bede-d46bba3095a8/volumes"
Sep 11 18:21:32.095231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:32.095193    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"]
Sep 11 18:21:32.099672 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:32.099646    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-794d1c-tnz7v"]
Sep 11 18:21:32.731256 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:32.731222    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0503417-229c-463e-9fa3-292d5316131c" path="/var/lib/kubelet/pods/c0503417-229c-463e-9fa3-292d5316131c/volumes"
Sep 11 18:21:39.111200 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.111164    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"]
Sep 11 18:21:39.111668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.111598    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy"
Sep 11 18:21:39.111668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.111610    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy"
Sep 11 18:21:39.111668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.111626    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0503417-229c-463e-9fa3-292d5316131c" containerName="revoke-keys"
Sep 11 18:21:39.111668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.111631    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0503417-229c-463e-9fa3-292d5316131c" containerName="revoke-keys"
Sep 11 18:21:39.111803 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.111694    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy"
Sep 11 18:21:39.111803 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.111703    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy"
Sep 11 18:21:39.111803 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.111711    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0503417-229c-463e-9fa3-292d5316131c" containerName="revoke-keys"
Sep 11 18:21:39.114715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.114697    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"
Sep 11 18:21:39.117394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.117373    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:21:39.123088 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.123022    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"]
Sep 11 18:21:39.219969 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.219929    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c69e85-f776-466d-b9de-aea313bcb195-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh\" (UID: \"09c69e85-f776-466d-b9de-aea313bcb195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"
Sep 11 18:21:39.320708 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.320672    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c69e85-f776-466d-b9de-aea313bcb195-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh\" (UID: \"09c69e85-f776-466d-b9de-aea313bcb195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"
Sep 11 18:21:39.321401 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.321376    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c69e85-f776-466d-b9de-aea313bcb195-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh\" (UID: \"09c69e85-f776-466d-b9de-aea313bcb195\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"
Sep 11 18:21:39.426969 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.426936    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"
Sep 11 18:21:39.697049 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.697022    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"]
Sep 11 18:21:39.697963 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:21:39.697938    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c69e85_f776_466d_b9de_aea313bcb195.slice/crio-478f21d2a62048a97ac1e856b4984c6bf3ff7ff7c0244e6aa3b52a1e789651a7 WatchSource:0}: Error finding container 478f21d2a62048a97ac1e856b4984c6bf3ff7ff7c0244e6aa3b52a1e789651a7: Status 404 returned error can't find the container with id 478f21d2a62048a97ac1e856b4984c6bf3ff7ff7c0244e6aa3b52a1e789651a7
Sep 11 18:21:39.778681 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:39.778637    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh" event={"ID":"09c69e85-f776-466d-b9de-aea313bcb195","Type":"ContainerStarted","Data":"478f21d2a62048a97ac1e856b4984c6bf3ff7ff7c0244e6aa3b52a1e789651a7"}
Sep 11 18:21:40.786555 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:40.786513    2571 generic.go:356] "Generic (PLEG): container finished" podID="09c69e85-f776-466d-b9de-aea313bcb195" containerID="387023819ef2c72ed2dbfa3b2caa18f7473d0279776cbb9e59a4321f71f6e65e" exitCode=0
Sep 11 18:21:40.787009 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:40.786596    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh" event={"ID":"09c69e85-f776-466d-b9de-aea313bcb195","Type":"ContainerDied","Data":"387023819ef2c72ed2dbfa3b2caa18f7473d0279776cbb9e59a4321f71f6e65e"}
Sep 11 18:21:41.934746 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:41.934718    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"
Sep 11 18:21:41.944493 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:41.944471    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c69e85-f776-466d-b9de-aea313bcb195-maas-api-service-ca\") pod \"09c69e85-f776-466d-b9de-aea313bcb195\" (UID: \"09c69e85-f776-466d-b9de-aea313bcb195\") "
Sep 11 18:21:41.944793 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:41.944772    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c69e85-f776-466d-b9de-aea313bcb195-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09c69e85-f776-466d-b9de-aea313bcb195" (UID: "09c69e85-f776-466d-b9de-aea313bcb195"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:42.045559 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:42.045477    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c69e85-f776-466d-b9de-aea313bcb195-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:42.800589 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:42.800555    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"
Sep 11 18:21:42.800783 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:42.800590    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh" event={"ID":"09c69e85-f776-466d-b9de-aea313bcb195","Type":"ContainerDied","Data":"478f21d2a62048a97ac1e856b4984c6bf3ff7ff7c0244e6aa3b52a1e789651a7"}
Sep 11 18:21:42.800783 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:42.800668    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="478f21d2a62048a97ac1e856b4984c6bf3ff7ff7c0244e6aa3b52a1e789651a7"
Sep 11 18:21:48.776627 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.776592    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"]
Sep 11 18:21:48.777100 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.777079    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy"
Sep 11 18:21:48.777100 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.777098    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e120ce16-8fa8-45e4-bede-d46bba3095a8" containerName="istio-proxy"
Sep 11 18:21:48.777267 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.777111    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c69e85-f776-466d-b9de-aea313bcb195" containerName="revoke-keys"
Sep 11 18:21:48.777267 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.777124    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c69e85-f776-466d-b9de-aea313bcb195" containerName="revoke-keys"
Sep 11 18:21:48.777267 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.777235    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c69e85-f776-466d-b9de-aea313bcb195" containerName="revoke-keys"
Sep 11 18:21:48.780474 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.780459    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"
Sep 11 18:21:48.783060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.783043    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:21:48.798195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.798168    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"]
Sep 11 18:21:48.808475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.808448    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc331e35-1338-4a3d-9b1f-0f2dc4029f34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz\" (UID: \"dc331e35-1338-4a3d-9b1f-0f2dc4029f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"
Sep 11 18:21:48.909720 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.909683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc331e35-1338-4a3d-9b1f-0f2dc4029f34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz\" (UID: \"dc331e35-1338-4a3d-9b1f-0f2dc4029f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"
Sep 11 18:21:48.910342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:48.910291    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc331e35-1338-4a3d-9b1f-0f2dc4029f34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz\" (UID: \"dc331e35-1338-4a3d-9b1f-0f2dc4029f34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"
Sep 11 18:21:49.102199 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.102113    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"
Sep 11 18:21:49.372676 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.372653    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"]
Sep 11 18:21:49.373737 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:21:49.373713    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc331e35_1338_4a3d_9b1f_0f2dc4029f34.slice/crio-a3e457cad31140ddcf9ff559572e79ebc1b58b9b73f076ea4ada30badafd3778 WatchSource:0}: Error finding container a3e457cad31140ddcf9ff559572e79ebc1b58b9b73f076ea4ada30badafd3778: Status 404 returned error can't find the container with id a3e457cad31140ddcf9ff559572e79ebc1b58b9b73f076ea4ada30badafd3778
Sep 11 18:21:49.849635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.849463    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz" event={"ID":"dc331e35-1338-4a3d-9b1f-0f2dc4029f34","Type":"ContainerStarted","Data":"f1d3ed1246e5bbda9eb062c38ef4fe606ddb14ab1f731b73d6a1404e78648b0f"}
Sep 11 18:21:49.849635 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.849532    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz" event={"ID":"dc331e35-1338-4a3d-9b1f-0f2dc4029f34","Type":"ContainerStarted","Data":"a3e457cad31140ddcf9ff559572e79ebc1b58b9b73f076ea4ada30badafd3778"}
Sep 11 18:21:49.878033 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.877984    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz" podStartSLOduration=1.877968461 podStartE2EDuration="1.877968461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:49.876096419 +0000 UTC m=+1673.646162768" watchObservedRunningTime="2026-09-11 18:21:49.877968461 +0000 UTC m=+1673.648034809"
Sep 11 18:21:49.919110 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.919042    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"]
Sep 11 18:21:49.919290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.919257    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz" podUID="24dc2906-ff7e-47f3-ad65-a8b1e984296b" containerName="payload-processing" containerID="cri-o://a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2" gracePeriod=30
Sep 11 18:21:49.957627 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.956868    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"]
Sep 11 18:21:49.957627 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:49.957107    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k" podUID="c5f710b2-8a94-4c79-bf2b-6c0553fe6169" containerName="payload-pre-processing" containerID="cri-o://36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17" gracePeriod=30
Sep 11 18:21:50.188935 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.188841    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"]
Sep 11 18:21:50.191884 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.191847    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b05610-bbjrh"]
Sep 11 18:21:50.244047 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.244026    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:21:50.251778 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.251754    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:21:50.323818 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.323789    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-plugins-config-volume\") pod \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\" (UID: \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\") "
Sep 11 18:21:50.324024 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.323875    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dc2906-ff7e-47f3-ad65-a8b1e984296b-plugins-config-volume\") pod \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\" (UID: \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\") "
Sep 11 18:21:50.324024 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.323915    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nvcb\" (UniqueName: \"kubernetes.io/projected/24dc2906-ff7e-47f3-ad65-a8b1e984296b-kube-api-access-9nvcb\") pod \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\" (UID: \"24dc2906-ff7e-47f3-ad65-a8b1e984296b\") "
Sep 11 18:21:50.324024 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.323988    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k99jh\" (UniqueName: \"kubernetes.io/projected/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-kube-api-access-k99jh\") pod \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\" (UID: \"c5f710b2-8a94-4c79-bf2b-6c0553fe6169\") "
Sep 11 18:21:50.324247 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.324225    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5f710b2-8a94-4c79-bf2b-6c0553fe6169" (UID: "c5f710b2-8a94-4c79-bf2b-6c0553fe6169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:50.325135 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.325104    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24dc2906-ff7e-47f3-ad65-a8b1e984296b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24dc2906-ff7e-47f3-ad65-a8b1e984296b" (UID: "24dc2906-ff7e-47f3-ad65-a8b1e984296b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:50.326353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.326277    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-kube-api-access-k99jh" (OuterVolumeSpecName: "kube-api-access-k99jh") pod "c5f710b2-8a94-4c79-bf2b-6c0553fe6169" (UID: "c5f710b2-8a94-4c79-bf2b-6c0553fe6169"). InnerVolumeSpecName "kube-api-access-k99jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:50.326798 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.326775    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24dc2906-ff7e-47f3-ad65-a8b1e984296b-kube-api-access-9nvcb" (OuterVolumeSpecName: "kube-api-access-9nvcb") pod "24dc2906-ff7e-47f3-ad65-a8b1e984296b" (UID: "24dc2906-ff7e-47f3-ad65-a8b1e984296b"). InnerVolumeSpecName "kube-api-access-9nvcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:50.425670 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.425639    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k99jh\" (UniqueName: \"kubernetes.io/projected/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-kube-api-access-k99jh\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:50.425670 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.425664    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5f710b2-8a94-4c79-bf2b-6c0553fe6169-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:50.425670 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.425674    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dc2906-ff7e-47f3-ad65-a8b1e984296b-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:50.425912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.425682    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nvcb\" (UniqueName: \"kubernetes.io/projected/24dc2906-ff7e-47f3-ad65-a8b1e984296b-kube-api-access-9nvcb\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:50.731133 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.731099    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c69e85-f776-466d-b9de-aea313bcb195" path="/var/lib/kubelet/pods/09c69e85-f776-466d-b9de-aea313bcb195/volumes"
Sep 11 18:21:50.858506 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.858473    2571 generic.go:356] "Generic (PLEG): container finished" podID="dc331e35-1338-4a3d-9b1f-0f2dc4029f34" containerID="f1d3ed1246e5bbda9eb062c38ef4fe606ddb14ab1f731b73d6a1404e78648b0f" exitCode=0
Sep 11 18:21:50.858888 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.858521    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz" event={"ID":"dc331e35-1338-4a3d-9b1f-0f2dc4029f34","Type":"ContainerDied","Data":"f1d3ed1246e5bbda9eb062c38ef4fe606ddb14ab1f731b73d6a1404e78648b0f"}
Sep 11 18:21:50.860008 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.859989    2571 generic.go:356] "Generic (PLEG): container finished" podID="c5f710b2-8a94-4c79-bf2b-6c0553fe6169" containerID="36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17" exitCode=0
Sep 11 18:21:50.860114 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.860036    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"
Sep 11 18:21:50.860114 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.860075    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k" event={"ID":"c5f710b2-8a94-4c79-bf2b-6c0553fe6169","Type":"ContainerDied","Data":"36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17"}
Sep 11 18:21:50.860227 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.860119    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k" event={"ID":"c5f710b2-8a94-4c79-bf2b-6c0553fe6169","Type":"ContainerDied","Data":"a1123c60cc0ff813839b5f31655507a0e8ef10fc1246bf0847231f80141275c5"}
Sep 11 18:21:50.860227 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.860149    2571 scope.go:117] "RemoveContainer" containerID="36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17"
Sep 11 18:21:50.861575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.861551    2571 generic.go:356] "Generic (PLEG): container finished" podID="24dc2906-ff7e-47f3-ad65-a8b1e984296b" containerID="a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2" exitCode=0
Sep 11 18:21:50.861685 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.861590    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"
Sep 11 18:21:50.861685 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.861636    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz" event={"ID":"24dc2906-ff7e-47f3-ad65-a8b1e984296b","Type":"ContainerDied","Data":"a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2"}
Sep 11 18:21:50.861795 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.861687    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz" event={"ID":"24dc2906-ff7e-47f3-ad65-a8b1e984296b","Type":"ContainerDied","Data":"e3122099719d98d57527f0eecec5de19ac5a523b74aa85614bbaba8fb84b2dc9"}
Sep 11 18:21:50.869886 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.869872    2571 scope.go:117] "RemoveContainer" containerID="36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17"
Sep 11 18:21:50.870142 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:50.870126    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17\": container with ID starting with 36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17 not found: ID does not exist" containerID="36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17"
Sep 11 18:21:50.870187 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.870149    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17"} err="failed to get container status \"36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17\": rpc error: code = NotFound desc = could not find container \"36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17\": container with ID starting with 36583b0252569a55d799452a73fefe5249019376e2035e484c60ac4a687f7a17 not found: ID does not exist"
Sep 11 18:21:50.870187 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.870165    2571 scope.go:117] "RemoveContainer" containerID="a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2"
Sep 11 18:21:50.878885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.878872    2571 scope.go:117] "RemoveContainer" containerID="a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2"
Sep 11 18:21:50.879108 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:21:50.879089    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2\": container with ID starting with a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2 not found: ID does not exist" containerID="a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2"
Sep 11 18:21:50.879160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.879115    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2"} err="failed to get container status \"a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2\": rpc error: code = NotFound desc = could not find container \"a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2\": container with ID starting with a73a68f0bff143a6641c05339243d06c4e0be4e95031a47c654178c24e600be2 not found: ID does not exist"
Sep 11 18:21:50.890315 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.890281    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"]
Sep 11 18:21:50.892901 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.892880    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b05610-6b85b9d889-hnsfz"]
Sep 11 18:21:50.904556 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.904527    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"]
Sep 11 18:21:50.908345 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:50.908325    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b05610-74fd7b75db-vsc2k"]
Sep 11 18:21:52.009483 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.009451    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"
Sep 11 18:21:52.040269 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.040238    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc331e35-1338-4a3d-9b1f-0f2dc4029f34-maas-api-service-ca\") pod \"dc331e35-1338-4a3d-9b1f-0f2dc4029f34\" (UID: \"dc331e35-1338-4a3d-9b1f-0f2dc4029f34\") "
Sep 11 18:21:52.040643 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.040625    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc331e35-1338-4a3d-9b1f-0f2dc4029f34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc331e35-1338-4a3d-9b1f-0f2dc4029f34" (UID: "dc331e35-1338-4a3d-9b1f-0f2dc4029f34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:52.141324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.141223    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc331e35-1338-4a3d-9b1f-0f2dc4029f34-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:21:52.731685 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.731654    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24dc2906-ff7e-47f3-ad65-a8b1e984296b" path="/var/lib/kubelet/pods/24dc2906-ff7e-47f3-ad65-a8b1e984296b/volumes"
Sep 11 18:21:52.732039 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.732026    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f710b2-8a94-4c79-bf2b-6c0553fe6169" path="/var/lib/kubelet/pods/c5f710b2-8a94-4c79-bf2b-6c0553fe6169/volumes"
Sep 11 18:21:52.879104 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.879076    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"
Sep 11 18:21:52.879272 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.879116    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz" event={"ID":"dc331e35-1338-4a3d-9b1f-0f2dc4029f34","Type":"ContainerDied","Data":"a3e457cad31140ddcf9ff559572e79ebc1b58b9b73f076ea4ada30badafd3778"}
Sep 11 18:21:52.879272 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:52.879170    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e457cad31140ddcf9ff559572e79ebc1b58b9b73f076ea4ada30badafd3778"
Sep 11 18:21:53.647917 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:53.647880    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"]
Sep 11 18:21:57.189455 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.189411    2571 scope.go:117] "RemoveContainer" containerID="933236af869f0b6afbe0ebcf514b725574fcbcd2acfa599ddd967b5dca643240"
Sep 11 18:21:57.202720 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.202691    2571 scope.go:117] "RemoveContainer" containerID="e775e5607a9c13ca75f11a7e8569a43215e0aa8ea6b6ddce5d8920dab36f8a29"
Sep 11 18:21:57.213216 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.213194    2571 scope.go:117] "RemoveContainer" containerID="3cd1ad1e4c1682699437f9fe478f118c76fa8852bc8fe3962327dbdbd02fe6c1"
Sep 11 18:21:57.221188 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.221170    2571 scope.go:117] "RemoveContainer" containerID="9cfbd6f1db754609f8696dcb657ca6d833023727ddad3e25650843d354fe70bb"
Sep 11 18:21:57.426898 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.426864    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"]
Sep 11 18:21:57.427280 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427266    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc331e35-1338-4a3d-9b1f-0f2dc4029f34" containerName="revoke-keys"
Sep 11 18:21:57.427375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427282    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc331e35-1338-4a3d-9b1f-0f2dc4029f34" containerName="revoke-keys"
Sep 11 18:21:57.427375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427315    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f710b2-8a94-4c79-bf2b-6c0553fe6169" containerName="payload-pre-processing"
Sep 11 18:21:57.427375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427321    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f710b2-8a94-4c79-bf2b-6c0553fe6169" containerName="payload-pre-processing"
Sep 11 18:21:57.427375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427328    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24dc2906-ff7e-47f3-ad65-a8b1e984296b" containerName="payload-processing"
Sep 11 18:21:57.427375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427333    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="24dc2906-ff7e-47f3-ad65-a8b1e984296b" containerName="payload-processing"
Sep 11 18:21:57.427557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427390    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f710b2-8a94-4c79-bf2b-6c0553fe6169" containerName="payload-pre-processing"
Sep 11 18:21:57.427557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427397    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc331e35-1338-4a3d-9b1f-0f2dc4029f34" containerName="revoke-keys"
Sep 11 18:21:57.427557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.427405    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="24dc2906-ff7e-47f3-ad65-a8b1e984296b" containerName="payload-processing"
Sep 11 18:21:57.432178 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.432161    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"
Sep 11 18:21:57.436046 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.436023    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:21:57.453844 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.453781    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"]
Sep 11 18:21:57.491885 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.491862    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d662beef-b2bc-4129-b25c-aae6607869f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh\" (UID: \"d662beef-b2bc-4129-b25c-aae6607869f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"
Sep 11 18:21:57.592358 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.592320    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d662beef-b2bc-4129-b25c-aae6607869f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh\" (UID: \"d662beef-b2bc-4129-b25c-aae6607869f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"
Sep 11 18:21:57.592969 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.592951    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d662beef-b2bc-4129-b25c-aae6607869f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh\" (UID: \"d662beef-b2bc-4129-b25c-aae6607869f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"
Sep 11 18:21:57.742347 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:57.742247    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"
Sep 11 18:21:58.015838 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:58.015811    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"]
Sep 11 18:21:58.017084 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:21:58.017057    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd662beef_b2bc_4129_b25c_aae6607869f3.slice/crio-dd55eb39920af83f34517f0396339797a8f7e2b954df7b2867f13d309b89c462 WatchSource:0}: Error finding container dd55eb39920af83f34517f0396339797a8f7e2b954df7b2867f13d309b89c462: Status 404 returned error can't find the container with id dd55eb39920af83f34517f0396339797a8f7e2b954df7b2867f13d309b89c462
Sep 11 18:21:58.925312 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:58.925238    2571 generic.go:356] "Generic (PLEG): container finished" podID="d662beef-b2bc-4129-b25c-aae6607869f3" containerID="113aafade582efcb3a04f180352cac619dbbc0b94aef0fd82d6ba03d50b70936" exitCode=0
Sep 11 18:21:58.925758 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:58.925332    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh" event={"ID":"d662beef-b2bc-4129-b25c-aae6607869f3","Type":"ContainerDied","Data":"113aafade582efcb3a04f180352cac619dbbc0b94aef0fd82d6ba03d50b70936"}
Sep 11 18:21:58.925758 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:21:58.925379    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh" event={"ID":"d662beef-b2bc-4129-b25c-aae6607869f3","Type":"ContainerStarted","Data":"dd55eb39920af83f34517f0396339797a8f7e2b954df7b2867f13d309b89c462"}
Sep 11 18:22:00.072756 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:00.072733    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"
Sep 11 18:22:00.215380 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:00.215268    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d662beef-b2bc-4129-b25c-aae6607869f3-maas-api-service-ca\") pod \"d662beef-b2bc-4129-b25c-aae6607869f3\" (UID: \"d662beef-b2bc-4129-b25c-aae6607869f3\") "
Sep 11 18:22:00.215686 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:00.215659    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d662beef-b2bc-4129-b25c-aae6607869f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d662beef-b2bc-4129-b25c-aae6607869f3" (UID: "d662beef-b2bc-4129-b25c-aae6607869f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:22:00.316204 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:00.316174    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d662beef-b2bc-4129-b25c-aae6607869f3-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:00.939029 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:00.939000    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"
Sep 11 18:22:00.939172 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:00.939031    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh" event={"ID":"d662beef-b2bc-4129-b25c-aae6607869f3","Type":"ContainerDied","Data":"dd55eb39920af83f34517f0396339797a8f7e2b954df7b2867f13d309b89c462"}
Sep 11 18:22:00.939172 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:00.939071    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd55eb39920af83f34517f0396339797a8f7e2b954df7b2867f13d309b89c462"
Sep 11 18:22:04.544125 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:04.544093    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"]
Sep 11 18:22:04.554802 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:04.554767    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4e0b0b-29819175-pskzd"]
Sep 11 18:22:04.730430 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:04.730400    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d" path="/var/lib/kubelet/pods/7ed94cfb-b010-4aed-bd1b-0e2bfaf8544d/volumes"
Sep 11 18:22:04.817924 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:04.817840    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"]
Sep 11 18:22:04.820325 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:04.820285    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4e0b0b-tc6jz"]
Sep 11 18:22:06.736104 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:06.736074    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc331e35-1338-4a3d-9b1f-0f2dc4029f34" path="/var/lib/kubelet/pods/dc331e35-1338-4a3d-9b1f-0f2dc4029f34/volumes"
Sep 11 18:22:08.528257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.528217    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"]
Sep 11 18:22:08.530795 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.530768    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-90057a-29819175-n9qrq"]
Sep 11 18:22:08.667291 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.667236    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl" podUID="11e8865e-ae35-48dc-b3b8-0b5538e16d61" containerName="main" containerID="cri-o://21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e" gracePeriod=45
Sep 11 18:22:08.723095 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.723055    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"]
Sep 11 18:22:08.730402 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.730363    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b5f6bf-694f-4238-91bc-c0efc6492347" path="/var/lib/kubelet/pods/76b5f6bf-694f-4238-91bc-c0efc6492347/volumes"
Sep 11 18:22:08.730651 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.730637    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-90057a-l8vnh"]
Sep 11 18:22:08.945364 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.945335    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:22:08.995127 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.995091    2571 generic.go:356] "Generic (PLEG): container finished" podID="11e8865e-ae35-48dc-b3b8-0b5538e16d61" containerID="21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e" exitCode=0
Sep 11 18:22:08.995321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.995139    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"
Sep 11 18:22:08.995321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.995149    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl" event={"ID":"11e8865e-ae35-48dc-b3b8-0b5538e16d61","Type":"ContainerDied","Data":"21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e"}
Sep 11 18:22:08.995321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.995200    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl" event={"ID":"11e8865e-ae35-48dc-b3b8-0b5538e16d61","Type":"ContainerDied","Data":"ee62c01a3affdac51da137371158ecc3228b1d0be4ecb0da16f3bb82ed75349b"}
Sep 11 18:22:08.995321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:08.995218    2571 scope.go:117] "RemoveContainer" containerID="21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e"
Sep 11 18:22:09.006432 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.006407    2571 scope.go:117] "RemoveContainer" containerID="21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e"
Sep 11 18:22:09.006738 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:22:09.006715    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e\": container with ID starting with 21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e not found: ID does not exist" containerID="21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e"
Sep 11 18:22:09.006834 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.006753    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e"} err="failed to get container status \"21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e\": rpc error: code = NotFound desc = could not find container \"21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e\": container with ID starting with 21d2bd18a4bf3fd969dda03248cfa11fca6bc2ed084ef5dca90d6b56bac5ea1e not found: ID does not exist"
Sep 11 18:22:09.103695 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.103608    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwk2g\" (UniqueName: \"kubernetes.io/projected/11e8865e-ae35-48dc-b3b8-0b5538e16d61-kube-api-access-dwk2g\") pod \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") "
Sep 11 18:22:09.103695 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.103690    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tls-certs\") pod \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") "
Sep 11 18:22:09.103921 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.103710    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-home\") pod \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") "
Sep 11 18:22:09.103921 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.103731    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tmp-dir\") pod \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") "
Sep 11 18:22:09.103921 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.103760    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-dshm\") pod \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") "
Sep 11 18:22:09.103921 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.103811    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-model-cache\") pod \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\" (UID: \"11e8865e-ae35-48dc-b3b8-0b5538e16d61\") "
Sep 11 18:22:09.104150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.104027    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-home" (OuterVolumeSpecName: "home") pod "11e8865e-ae35-48dc-b3b8-0b5538e16d61" (UID: "11e8865e-ae35-48dc-b3b8-0b5538e16d61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.104150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.104014    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11e8865e-ae35-48dc-b3b8-0b5538e16d61" (UID: "11e8865e-ae35-48dc-b3b8-0b5538e16d61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.104150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.104089    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-model-cache" (OuterVolumeSpecName: "model-cache") pod "11e8865e-ae35-48dc-b3b8-0b5538e16d61" (UID: "11e8865e-ae35-48dc-b3b8-0b5538e16d61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.105781 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.105753    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-dshm" (OuterVolumeSpecName: "dshm") pod "11e8865e-ae35-48dc-b3b8-0b5538e16d61" (UID: "11e8865e-ae35-48dc-b3b8-0b5538e16d61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.105781 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.105763    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e8865e-ae35-48dc-b3b8-0b5538e16d61-kube-api-access-dwk2g" (OuterVolumeSpecName: "kube-api-access-dwk2g") pod "11e8865e-ae35-48dc-b3b8-0b5538e16d61" (UID: "11e8865e-ae35-48dc-b3b8-0b5538e16d61"). InnerVolumeSpecName "kube-api-access-dwk2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:09.105956 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.105796    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11e8865e-ae35-48dc-b3b8-0b5538e16d61" (UID: "11e8865e-ae35-48dc-b3b8-0b5538e16d61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:22:09.205247 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.205208    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.205247 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.205244    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.205247 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.205252    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.205247 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.205260    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.205247 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.205269    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11e8865e-ae35-48dc-b3b8-0b5538e16d61-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.205561 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.205278    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwk2g\" (UniqueName: \"kubernetes.io/projected/11e8865e-ae35-48dc-b3b8-0b5538e16d61-kube-api-access-dwk2g\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.317323 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.317271    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"]
Sep 11 18:22:09.322029 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:09.322007    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-2-w4-4e0b0b-kserve-56d86c85dc-97nxl"]
Sep 11 18:22:10.731997 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:10.731956    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11e8865e-ae35-48dc-b3b8-0b5538e16d61" path="/var/lib/kubelet/pods/11e8865e-ae35-48dc-b3b8-0b5538e16d61/volumes"
Sep 11 18:22:10.732623 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:10.732488    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d662beef-b2bc-4129-b25c-aae6607869f3" path="/var/lib/kubelet/pods/d662beef-b2bc-4129-b25c-aae6607869f3/volumes"
Sep 11 18:22:12.068687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:12.068646    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"]
Sep 11 18:22:12.069085 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:12.068910    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy" containerID="cri-o://ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27" gracePeriod=30
Sep 11 18:22:14.613898 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:14.613854    2571 patch_prober.go:28] interesting pod/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:22:14.614279 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:14.613922    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:22:15.559390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:15.559356    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"]
Sep 11 18:22:15.559605 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:15.559574    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy" containerID="cri-o://3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d" gracePeriod=30
Sep 11 18:22:17.329324 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.329285    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:22:17.480114 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480078    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dnj2\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-kube-api-access-7dnj2\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480147    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-envoy\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480183    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-certs\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480362    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc14be55-d6a6-4ec1-8c65-9057f026741a-istiod-ca-cert\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480412    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-podinfo\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480426    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:17.480523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480443    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-data\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480477    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-socket\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480523 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480501    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-token\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480539    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-credential-socket\") pod \"dc14be55-d6a6-4ec1-8c65-9057f026741a\" (UID: \"dc14be55-d6a6-4ec1-8c65-9057f026741a\") "
Sep 11 18:22:17.480843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480700    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:17.480843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480716    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:17.480843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480762    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc14be55-d6a6-4ec1-8c65-9057f026741a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:22:17.481061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.480886    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:17.481773 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.481734    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:17.481890 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.481783    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:17.481890 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.481811    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:17.481890 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.481849    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc14be55-d6a6-4ec1-8c65-9057f026741a-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:17.481890 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.481866    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:17.483171 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.482758    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:22:17.490145 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.490110    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:17.490237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.490168    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:17.490237 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.490181    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-kube-api-access-7dnj2" (OuterVolumeSpecName: "kube-api-access-7dnj2") pod "dc14be55-d6a6-4ec1-8c65-9057f026741a" (UID: "dc14be55-d6a6-4ec1-8c65-9057f026741a"). InnerVolumeSpecName "kube-api-access-7dnj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:17.582947 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.582909    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:17.582947 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.582939    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:17.582947 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.582949    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dnj2\" (UniqueName: \"kubernetes.io/projected/dc14be55-d6a6-4ec1-8c65-9057f026741a-kube-api-access-7dnj2\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:17.583326 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:17.582959    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc14be55-d6a6-4ec1-8c65-9057f026741a-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:18.059934 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.059889    2571 generic.go:356] "Generic (PLEG): container finished" podID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerID="ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27" exitCode=0
Sep 11 18:22:18.060117 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.059948    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" event={"ID":"dc14be55-d6a6-4ec1-8c65-9057f026741a","Type":"ContainerDied","Data":"ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27"}
Sep 11 18:22:18.060117 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.059969    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"
Sep 11 18:22:18.060117 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.060003    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds" event={"ID":"dc14be55-d6a6-4ec1-8c65-9057f026741a","Type":"ContainerDied","Data":"dd160d8e3fb715d61ce0ae9a021e1a81b0e65d7a833a691a9d95e5440928e404"}
Sep 11 18:22:18.060117 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.060031    2571 scope.go:117] "RemoveContainer" containerID="ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27"
Sep 11 18:22:18.070057 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.070040    2571 scope.go:117] "RemoveContainer" containerID="2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0"
Sep 11 18:22:18.078639 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.078619    2571 scope.go:117] "RemoveContainer" containerID="ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27"
Sep 11 18:22:18.078906 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:22:18.078889    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27\": container with ID starting with ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27 not found: ID does not exist" containerID="ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27"
Sep 11 18:22:18.078983 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.078913    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27"} err="failed to get container status \"ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27\": rpc error: code = NotFound desc = could not find container \"ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27\": container with ID starting with ea4ab52faab8ae223383781ad36141fac40734b9894d285b95dd59a3f489dc27 not found: ID does not exist"
Sep 11 18:22:18.078983 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.078931    2571 scope.go:117] "RemoveContainer" containerID="2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0"
Sep 11 18:22:18.079200 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:22:18.079186    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0\": container with ID starting with 2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0 not found: ID does not exist" containerID="2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0"
Sep 11 18:22:18.079242 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.079203    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0"} err="failed to get container status \"2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0\": rpc error: code = NotFound desc = could not find container \"2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0\": container with ID starting with 2bd58737d19bb039befc525a2b5b6bbe4937f8c83b984fd755014510e9e130d0 not found: ID does not exist"
Sep 11 18:22:18.085460 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.085436    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"]
Sep 11 18:22:18.089709 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.089688    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4e0b0b-openshift-default-6f58f6fd5d-679ds"]
Sep 11 18:22:18.376907 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.376829    2571 patch_prober.go:28] interesting pod/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:22:18.377248 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.376902    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:22:18.730992 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:18.730951    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" path="/var/lib/kubelet/pods/dc14be55-d6a6-4ec1-8c65-9057f026741a/volumes"
Sep 11 18:22:20.808686 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.808665    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:22:20.915558 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915533    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-token\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.915705 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915567    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-data\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.915705 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915598    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-envoy\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.915705 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915674    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-certs\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.915830 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915711    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istiod-ca-cert\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.915830 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915740    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-socket\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.915830 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915773    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-podinfo\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.915830 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915800    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-credential-socket\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.916005 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915862    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vjs4\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-kube-api-access-6vjs4\") pod \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\" (UID: \"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef\") "
Sep 11 18:22:20.916005 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915867    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:20.916005 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.915954    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:20.916167 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.916093    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:20.916220 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.916201    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:22:20.916273 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.916228    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:20.916273 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.916247    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:20.916273 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.916260    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:20.916464 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.916356    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:20.917858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.917829    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:20.917968 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.917895    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:20.918048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.918027    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:22:20.918285 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:20.918267    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-kube-api-access-6vjs4" (OuterVolumeSpecName: "kube-api-access-6vjs4") pod "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" (UID: "3c238199-9a97-4c3d-a5d2-a0d7d367c7ef"). InnerVolumeSpecName "kube-api-access-6vjs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:21.017228 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.017199    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vjs4\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-kube-api-access-6vjs4\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:21.017228 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.017228    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:21.017228 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.017237    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:21.017459 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.017244    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:21.017459 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.017252    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:21.017459 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.017260    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:21.082059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.082024    2571 generic.go:356] "Generic (PLEG): container finished" podID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerID="3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d" exitCode=0
Sep 11 18:22:21.082226 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.082073    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"
Sep 11 18:22:21.082226 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.082093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" event={"ID":"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef","Type":"ContainerDied","Data":"3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d"}
Sep 11 18:22:21.082226 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.082144    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4" event={"ID":"3c238199-9a97-4c3d-a5d2-a0d7d367c7ef","Type":"ContainerDied","Data":"ffff6d7209a27e5d7fbc3afe2294e11b9667169be38f1581f143773268c7b5b5"}
Sep 11 18:22:21.082226 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.082168    2571 scope.go:117] "RemoveContainer" containerID="3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d"
Sep 11 18:22:21.092693 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.092675    2571 scope.go:117] "RemoveContainer" containerID="125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee"
Sep 11 18:22:21.101042 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.101024    2571 scope.go:117] "RemoveContainer" containerID="3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d"
Sep 11 18:22:21.101330 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:22:21.101311    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d\": container with ID starting with 3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d not found: ID does not exist" containerID="3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d"
Sep 11 18:22:21.101380 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.101338    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d"} err="failed to get container status \"3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d\": rpc error: code = NotFound desc = could not find container \"3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d\": container with ID starting with 3f14df9ba210de818f1bfd9dcea237a7465c11601c54de3589de702394d2d85d not found: ID does not exist"
Sep 11 18:22:21.101380 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.101355    2571 scope.go:117] "RemoveContainer" containerID="125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee"
Sep 11 18:22:21.101592 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:22:21.101577    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee\": container with ID starting with 125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee not found: ID does not exist" containerID="125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee"
Sep 11 18:22:21.101628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.101596    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee"} err="failed to get container status \"125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee\": rpc error: code = NotFound desc = could not find container \"125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee\": container with ID starting with 125036187dd44c0f196503bd43813a52e8c4c5498b9976f5b368f7586d4e9dee not found: ID does not exist"
Sep 11 18:22:21.106255 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.106233    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"]
Sep 11 18:22:21.110716 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:21.110692    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-90057a-openshift-default-598cc5f744-5fpg4"]
Sep 11 18:22:22.730568 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:22.730533    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" path="/var/lib/kubelet/pods/3c238199-9a97-4c3d-a5d2-a0d7d367c7ef/volumes"
Sep 11 18:22:35.137867 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.137832    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7499-e2f6eb"]
Sep 11 18:22:35.138442 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138423    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy"
Sep 11 18:22:35.138529 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138445    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy"
Sep 11 18:22:35.138529 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138458    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy"
Sep 11 18:22:35.138529 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138466    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy"
Sep 11 18:22:35.138529 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138489    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11e8865e-ae35-48dc-b3b8-0b5538e16d61" containerName="main"
Sep 11 18:22:35.138529 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138498    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="11e8865e-ae35-48dc-b3b8-0b5538e16d61" containerName="main"
Sep 11 18:22:35.138837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138534    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d662beef-b2bc-4129-b25c-aae6607869f3" containerName="revoke-keys"
Sep 11 18:22:35.138837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138542    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d662beef-b2bc-4129-b25c-aae6607869f3" containerName="revoke-keys"
Sep 11 18:22:35.138837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138634    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy"
Sep 11 18:22:35.138837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138647    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy"
Sep 11 18:22:35.138837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138655    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy"
Sep 11 18:22:35.138837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138665    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="11e8865e-ae35-48dc-b3b8-0b5538e16d61" containerName="main"
Sep 11 18:22:35.138837 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.138674    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d662beef-b2bc-4129-b25c-aae6607869f3" containerName="revoke-keys"
Sep 11 18:22:35.141972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.141952    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-e2f6eb"
Sep 11 18:22:35.144733 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.144711    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q6zqg\""
Sep 11 18:22:35.145368 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.145345    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7499-e2f6eb"]
Sep 11 18:22:35.238823 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.238794    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m89lc\" (UniqueName: \"kubernetes.io/projected/038acbc4-79a4-4933-bc69-ea9c15422d8d-kube-api-access-m89lc\") pod \"test-curl-7499-e2f6eb\" (UID: \"038acbc4-79a4-4933-bc69-ea9c15422d8d\") " pod="opendatahub/test-curl-7499-e2f6eb"
Sep 11 18:22:35.339355 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.339321    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89lc\" (UniqueName: \"kubernetes.io/projected/038acbc4-79a4-4933-bc69-ea9c15422d8d-kube-api-access-m89lc\") pod \"test-curl-7499-e2f6eb\" (UID: \"038acbc4-79a4-4933-bc69-ea9c15422d8d\") " pod="opendatahub/test-curl-7499-e2f6eb"
Sep 11 18:22:35.349128 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.349092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89lc\" (UniqueName: \"kubernetes.io/projected/038acbc4-79a4-4933-bc69-ea9c15422d8d-kube-api-access-m89lc\") pod \"test-curl-7499-e2f6eb\" (UID: \"038acbc4-79a4-4933-bc69-ea9c15422d8d\") " pod="opendatahub/test-curl-7499-e2f6eb"
Sep 11 18:22:35.452570 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.452539    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-e2f6eb"
Sep 11 18:22:35.721608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:35.721580    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7499-e2f6eb"]
Sep 11 18:22:35.723059 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:22:35.723027    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod038acbc4_79a4_4933_bc69_ea9c15422d8d.slice/crio-c7bbc15f0ac0edbd296e580a10689dc7b178cbb0f11b41ef309efc31805834d1 WatchSource:0}: Error finding container c7bbc15f0ac0edbd296e580a10689dc7b178cbb0f11b41ef309efc31805834d1: Status 404 returned error can't find the container with id c7bbc15f0ac0edbd296e580a10689dc7b178cbb0f11b41ef309efc31805834d1
Sep 11 18:22:36.188309 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:36.188261    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7499-e2f6eb" event={"ID":"038acbc4-79a4-4933-bc69-ea9c15422d8d","Type":"ContainerStarted","Data":"f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0"}
Sep 11 18:22:36.188674 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:36.188324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7499-e2f6eb" event={"ID":"038acbc4-79a4-4933-bc69-ea9c15422d8d","Type":"ContainerStarted","Data":"c7bbc15f0ac0edbd296e580a10689dc7b178cbb0f11b41ef309efc31805834d1"}
Sep 11 18:22:36.203975 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:36.203929    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7499-e2f6eb" podStartSLOduration=1.203913441 podStartE2EDuration="1.203913441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:36.20317257 +0000 UTC m=+1719.973238915" watchObservedRunningTime="2026-09-11 18:22:36.203913441 +0000 UTC m=+1719.973979814"
Sep 11 18:22:36.790616 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:36.790581    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7499-e2f6eb"]
Sep 11 18:22:36.794396 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:36.794369    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7499-e2f6eb"]
Sep 11 18:22:37.195158 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:22:37.195119    2571 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7499-e2f6eb" container="test-curl-7499-e2f6eb"
Sep 11 18:22:38.115816 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:38.115782    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"]
Sep 11 18:22:38.208264 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:38.203274    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7499-e2f6eb" podUID="038acbc4-79a4-4933-bc69-ea9c15422d8d" containerName="test-curl-7499-e2f6eb" containerID="cri-o://f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0" gracePeriod=2
Sep 11 18:22:38.208743 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:22:38.208683    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7499-e2f6eb\" is forbidden: User \"system:node:ip-10-0-133-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-109.ec2.internal' and this object" podUID="038acbc4-79a4-4933-bc69-ea9c15422d8d" pod="opendatahub/test-curl-7499-e2f6eb"
Sep 11 18:22:40.367484 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:40.367458    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-e2f6eb"
Sep 11 18:22:40.483363 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:40.483253    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m89lc\" (UniqueName: \"kubernetes.io/projected/038acbc4-79a4-4933-bc69-ea9c15422d8d-kube-api-access-m89lc\") pod \"038acbc4-79a4-4933-bc69-ea9c15422d8d\" (UID: \"038acbc4-79a4-4933-bc69-ea9c15422d8d\") "
Sep 11 18:22:40.485360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:40.485329    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038acbc4-79a4-4933-bc69-ea9c15422d8d-kube-api-access-m89lc" (OuterVolumeSpecName: "kube-api-access-m89lc") pod "038acbc4-79a4-4933-bc69-ea9c15422d8d" (UID: "038acbc4-79a4-4933-bc69-ea9c15422d8d"). InnerVolumeSpecName "kube-api-access-m89lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:40.584235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:40.584195    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m89lc\" (UniqueName: \"kubernetes.io/projected/038acbc4-79a4-4933-bc69-ea9c15422d8d-kube-api-access-m89lc\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:40.731478 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:40.731449    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="038acbc4-79a4-4933-bc69-ea9c15422d8d" path="/var/lib/kubelet/pods/038acbc4-79a4-4933-bc69-ea9c15422d8d/volumes"
Sep 11 18:22:41.224260 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:41.224222    2571 generic.go:356] "Generic (PLEG): container finished" podID="038acbc4-79a4-4933-bc69-ea9c15422d8d" containerID="f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0" exitCode=137
Sep 11 18:22:41.224431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:41.224266    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-e2f6eb"
Sep 11 18:22:41.224431 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:41.224310    2571 scope.go:117] "RemoveContainer" containerID="f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0"
Sep 11 18:22:41.233463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:41.233445    2571 scope.go:117] "RemoveContainer" containerID="f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0"
Sep 11 18:22:41.233753 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:22:41.233734    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0\": container with ID starting with f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0 not found: ID does not exist" containerID="f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0"
Sep 11 18:22:41.233802 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:41.233767    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0"} err="failed to get container status \"f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0\": rpc error: code = NotFound desc = could not find container \"f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0\": container with ID starting with f44f962c0b784a9e0b0afad38e0a6b0cb97d7b6d55cd7606bbe3987125af61c0 not found: ID does not exist"
Sep 11 18:22:44.505712 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.505671    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"]
Sep 11 18:22:44.506343 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.506323    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy"
Sep 11 18:22:44.506437 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.506346    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy"
Sep 11 18:22:44.506437 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.506360    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy"
Sep 11 18:22:44.506437 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.506369    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc14be55-d6a6-4ec1-8c65-9057f026741a" containerName="istio-proxy"
Sep 11 18:22:44.506437 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.506411    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="038acbc4-79a4-4933-bc69-ea9c15422d8d" containerName="test-curl-7499-e2f6eb"
Sep 11 18:22:44.506437 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.506421    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="038acbc4-79a4-4933-bc69-ea9c15422d8d" containerName="test-curl-7499-e2f6eb"
Sep 11 18:22:44.506687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.506552    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c238199-9a97-4c3d-a5d2-a0d7d367c7ef" containerName="istio-proxy"
Sep 11 18:22:44.506687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.506568    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="038acbc4-79a4-4933-bc69-ea9c15422d8d" containerName="test-curl-7499-e2f6eb"
Sep 11 18:22:44.511338 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.511318    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.514203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.514178    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"sub-test-model-a5bbf6-kserve-self-signed-certs\""
Sep 11 18:22:44.520461 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.520181    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"]
Sep 11 18:22:44.622331 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.622283    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-dshm\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.622621 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.622591    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f913a96-21ec-4d86-b96a-1eea7c86da25-tls-certs\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.622766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.622647    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-home\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.622766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.622691    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-model-cache\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.622932 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.622841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-tmp-dir\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.623042 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.623018    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rszkv\" (UniqueName: \"kubernetes.io/projected/8f913a96-21ec-4d86-b96a-1eea7c86da25-kube-api-access-rszkv\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724105    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-home\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724142    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-model-cache\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724439 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724173    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-tmp-dir\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724439 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rszkv\" (UniqueName: \"kubernetes.io/projected/8f913a96-21ec-4d86-b96a-1eea7c86da25-kube-api-access-rszkv\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724439 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724244    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-dshm\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724439 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724270    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f913a96-21ec-4d86-b96a-1eea7c86da25-tls-certs\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724661 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724590    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-home\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724661 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724637    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-model-cache\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.724757 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.724663    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-tmp-dir\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.726610 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.726588    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-dshm\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.726782 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.726764    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f913a96-21ec-4d86-b96a-1eea7c86da25-tls-certs\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.733910 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.733879    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rszkv\" (UniqueName: \"kubernetes.io/projected/8f913a96-21ec-4d86-b96a-1eea7c86da25-kube-api-access-rszkv\") pod \"sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:44.824085 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:44.824000    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:45.096221 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:45.096195    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"]
Sep 11 18:22:45.097687 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:22:45.097657    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f913a96_21ec_4d86_b96a_1eea7c86da25.slice/crio-033b2e45269d9ca238601949f4e93e7fa0aca583b9c6c216b5ed0273df03f146 WatchSource:0}: Error finding container 033b2e45269d9ca238601949f4e93e7fa0aca583b9c6c216b5ed0273df03f146: Status 404 returned error can't find the container with id 033b2e45269d9ca238601949f4e93e7fa0aca583b9c6c216b5ed0273df03f146
Sep 11 18:22:45.252762 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:45.252721    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg" event={"ID":"8f913a96-21ec-4d86-b96a-1eea7c86da25","Type":"ContainerStarted","Data":"4250b4a1ffb1aa789342e48ece620a9dbc66fee31978db7b1581cc0a01cb8652"}
Sep 11 18:22:45.252963 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:45.252776    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg" event={"ID":"8f913a96-21ec-4d86-b96a-1eea7c86da25","Type":"ContainerStarted","Data":"033b2e45269d9ca238601949f4e93e7fa0aca583b9c6c216b5ed0273df03f146"}
Sep 11 18:22:45.271784 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:45.271721    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg" podStartSLOduration=1.271697925 podStartE2EDuration="1.271697925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:45.270712751 +0000 UTC m=+1729.040779109" watchObservedRunningTime="2026-09-11 18:22:45.271697925 +0000 UTC m=+1729.041764274"
Sep 11 18:22:53.135096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.135048    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw" podUID="8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" containerName="main" containerID="cri-o://044145bdf15e35a551ceac9c882a57c7aafc0e7a5cea7598de2cd05994869e7f" gracePeriod=45
Sep 11 18:22:53.307605 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.307574    2571 generic.go:356] "Generic (PLEG): container finished" podID="8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" containerID="044145bdf15e35a551ceac9c882a57c7aafc0e7a5cea7598de2cd05994869e7f" exitCode=0
Sep 11 18:22:53.307793 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.307654    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw" event={"ID":"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce","Type":"ContainerDied","Data":"044145bdf15e35a551ceac9c882a57c7aafc0e7a5cea7598de2cd05994869e7f"}
Sep 11 18:22:53.387261 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.387198    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:22:53.505471 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505435    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tmp-dir\") pod \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") "
Sep 11 18:22:53.505471 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505476    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-model-cache\") pod \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") "
Sep 11 18:22:53.505687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505506    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-home\") pod \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") "
Sep 11 18:22:53.505687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505529    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-dshm\") pod \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") "
Sep 11 18:22:53.505687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505556    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tls-certs\") pod \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") "
Sep 11 18:22:53.505687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505616    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpz82\" (UniqueName: \"kubernetes.io/projected/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-kube-api-access-wpz82\") pod \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\" (UID: \"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce\") "
Sep 11 18:22:53.505893 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505732    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" (UID: "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:53.505893 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505773    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" (UID: "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:53.505893 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505790    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-home" (OuterVolumeSpecName: "home") pod "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" (UID: "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:53.506040 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505920    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:53.506040 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505962    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:53.506040 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.505975    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:53.507851 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.507825    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-dshm" (OuterVolumeSpecName: "dshm") pod "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" (UID: "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:53.507851 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.507827    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-kube-api-access-wpz82" (OuterVolumeSpecName: "kube-api-access-wpz82") pod "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" (UID: "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce"). InnerVolumeSpecName "kube-api-access-wpz82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:53.507979 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.507919    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" (UID: "8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:22:53.607129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.607093    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpz82\" (UniqueName: \"kubernetes.io/projected/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-kube-api-access-wpz82\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:53.607129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.607125    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:53.607129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:53.607134    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:22:54.314711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.314682    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"
Sep 11 18:22:54.314711 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.314692    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw" event={"ID":"8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce","Type":"ContainerDied","Data":"6d669cb2d98aa7079c95b7ae89c78e1e214feb0f50926e6a0f897d19cde278e7"}
Sep 11 18:22:54.315222 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.314737    2571 scope.go:117] "RemoveContainer" containerID="044145bdf15e35a551ceac9c882a57c7aafc0e7a5cea7598de2cd05994869e7f"
Sep 11 18:22:54.336926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.336897    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"]
Sep 11 18:22:54.339261 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.339239    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/auth-test-model-a5bbf6-kserve-747f6f86c5-xbvpw"]
Sep 11 18:22:54.730367 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.730333    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" path="/var/lib/kubelet/pods/8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce/volumes"
Sep 11 18:22:54.824187 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.824159    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:54.824378 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.824208    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:54.836473 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:54.836446    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:55.332353 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:55.332325    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:22:57.297102 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:57.297074    2571 scope.go:117] "RemoveContainer" containerID="65adf3b0dd72511624ad74909eb161fa97748b89de0e2ee04b6474192ec6f4a5"
Sep 11 18:22:57.306290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:22:57.306272    2571 scope.go:117] "RemoveContainer" containerID="4c4ed22d8d51dd93d41586c64d297948402cbc3913dfab223da75acb92e420e4"
Sep 11 18:23:18.222275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.222238    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"]
Sep 11 18:23:18.222784 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.222695    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" containerName="main"
Sep 11 18:23:18.222784 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.222710    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" containerName="main"
Sep 11 18:23:18.222784 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.222784    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab54ffe-0ee4-446e-a4a4-d2af3f4ffbce" containerName="main"
Sep 11 18:23:18.225601 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.225581    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.229822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.229799    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"sub-test-model-5d35e7-kserve-self-signed-certs\""
Sep 11 18:23:18.229949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.229799    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"default-dockercfg-bktdb\""
Sep 11 18:23:18.229949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.229802    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"kube-root-ca.crt\""
Sep 11 18:23:18.229949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.229841    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"openshift-service-ca.crt\""
Sep 11 18:23:18.238449 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.238419    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"]
Sep 11 18:23:18.326267 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.326229    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tls-certs\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.326483 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.326287    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7nz\" (UniqueName: \"kubernetes.io/projected/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-kube-api-access-rr7nz\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.326483 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.326362    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-model-cache\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.326483 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.326397    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tmp-dir\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.326483 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.326430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-dshm\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.326664 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.326502    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-home\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.427680 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.427644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-model-cache\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.427887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.427695    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tmp-dir\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.427887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.427728    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-dshm\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.427887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.427766    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-home\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.427887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.427801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tls-certs\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.427887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.427857    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7nz\" (UniqueName: \"kubernetes.io/projected/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-kube-api-access-rr7nz\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.428148 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.428119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-model-cache\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.428148 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.428142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tmp-dir\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.428250 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.428188    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-home\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.430017 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.429992    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-dshm\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.430168 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.430150    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tls-certs\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.436657 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.436638    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7nz\" (UniqueName: \"kubernetes.io/projected/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-kube-api-access-rr7nz\") pod \"sub-test-model-5d35e7-kserve-d78b764b5-qsrvl\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.534827 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.534757    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:18.811954 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:18.811923    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"]
Sep 11 18:23:18.812482 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:23:18.812457    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd249de2a_c06d_4a4b_b3fe_b7cbd5bdc032.slice/crio-5a8c724fde4113ec541c2017d99226cb65fd96026473061d73f1ce36b6af6ff9 WatchSource:0}: Error finding container 5a8c724fde4113ec541c2017d99226cb65fd96026473061d73f1ce36b6af6ff9: Status 404 returned error can't find the container with id 5a8c724fde4113ec541c2017d99226cb65fd96026473061d73f1ce36b6af6ff9
Sep 11 18:23:19.470358 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:19.470325    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl" event={"ID":"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032","Type":"ContainerStarted","Data":"a6970fe23d13646465faaa9f2d360876e731352f6c64fdbc61179d60dd05d3b0"}
Sep 11 18:23:19.470726 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:19.470370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl" event={"ID":"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032","Type":"ContainerStarted","Data":"5a8c724fde4113ec541c2017d99226cb65fd96026473061d73f1ce36b6af6ff9"}
Sep 11 18:23:19.491143 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:19.491095    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl" podStartSLOduration=1.491080888 podStartE2EDuration="1.491080888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:19.490187726 +0000 UTC m=+1763.260254085" watchObservedRunningTime="2026-09-11 18:23:19.491080888 +0000 UTC m=+1763.261147237"
Sep 11 18:23:28.535524 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:28.535489    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:28.535982 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:28.535559    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:28.547951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:28.547929    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:29.540967 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:29.540938    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:23:57.351581 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:23:57.351543    2571 scope.go:117] "RemoveContainer" containerID="d19f996f4de25fbd879bcf6a3bbcaac8875d069cc475b746fe050a993d97dcfe"
Sep 11 18:24:03.602575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:03.602540    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"]
Sep 11 18:24:04.644631 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:04.644596    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"]
Sep 11 18:24:18.621563 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.621489    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg" podUID="8f913a96-21ec-4d86-b96a-1eea7c86da25" containerName="main" containerID="cri-o://4250b4a1ffb1aa789342e48ece620a9dbc66fee31978db7b1581cc0a01cb8652" gracePeriod=45
Sep 11 18:24:18.830278 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.830247    2571 generic.go:356] "Generic (PLEG): container finished" podID="8f913a96-21ec-4d86-b96a-1eea7c86da25" containerID="4250b4a1ffb1aa789342e48ece620a9dbc66fee31978db7b1581cc0a01cb8652" exitCode=0
Sep 11 18:24:18.830470 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.830324    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg" event={"ID":"8f913a96-21ec-4d86-b96a-1eea7c86da25","Type":"ContainerDied","Data":"4250b4a1ffb1aa789342e48ece620a9dbc66fee31978db7b1581cc0a01cb8652"}
Sep 11 18:24:18.865320 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.865280    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:24:18.971714 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.971679    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-tmp-dir\") pod \"8f913a96-21ec-4d86-b96a-1eea7c86da25\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") "
Sep 11 18:24:18.971714 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.971713    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-home\") pod \"8f913a96-21ec-4d86-b96a-1eea7c86da25\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") "
Sep 11 18:24:18.971958 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.971741    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-dshm\") pod \"8f913a96-21ec-4d86-b96a-1eea7c86da25\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") "
Sep 11 18:24:18.971958 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.971798    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f913a96-21ec-4d86-b96a-1eea7c86da25-tls-certs\") pod \"8f913a96-21ec-4d86-b96a-1eea7c86da25\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") "
Sep 11 18:24:18.971958 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.971818    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-model-cache\") pod \"8f913a96-21ec-4d86-b96a-1eea7c86da25\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") "
Sep 11 18:24:18.971958 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.971849    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rszkv\" (UniqueName: \"kubernetes.io/projected/8f913a96-21ec-4d86-b96a-1eea7c86da25-kube-api-access-rszkv\") pod \"8f913a96-21ec-4d86-b96a-1eea7c86da25\" (UID: \"8f913a96-21ec-4d86-b96a-1eea7c86da25\") "
Sep 11 18:24:18.971958 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.971942    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f913a96-21ec-4d86-b96a-1eea7c86da25" (UID: "8f913a96-21ec-4d86-b96a-1eea7c86da25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:18.972216 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.971982    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-home" (OuterVolumeSpecName: "home") pod "8f913a96-21ec-4d86-b96a-1eea7c86da25" (UID: "8f913a96-21ec-4d86-b96a-1eea7c86da25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:18.972216 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.972149    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:18.972216 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.972183    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:18.972216 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.972157    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f913a96-21ec-4d86-b96a-1eea7c86da25" (UID: "8f913a96-21ec-4d86-b96a-1eea7c86da25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:18.973955 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.973927    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f913a96-21ec-4d86-b96a-1eea7c86da25-kube-api-access-rszkv" (OuterVolumeSpecName: "kube-api-access-rszkv") pod "8f913a96-21ec-4d86-b96a-1eea7c86da25" (UID: "8f913a96-21ec-4d86-b96a-1eea7c86da25"). InnerVolumeSpecName "kube-api-access-rszkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:18.974065 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.973956    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-dshm" (OuterVolumeSpecName: "dshm") pod "8f913a96-21ec-4d86-b96a-1eea7c86da25" (UID: "8f913a96-21ec-4d86-b96a-1eea7c86da25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:18.974065 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:18.973953    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f913a96-21ec-4d86-b96a-1eea7c86da25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f913a96-21ec-4d86-b96a-1eea7c86da25" (UID: "8f913a96-21ec-4d86-b96a-1eea7c86da25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:24:19.073115 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.073075    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rszkv\" (UniqueName: \"kubernetes.io/projected/8f913a96-21ec-4d86-b96a-1eea7c86da25-kube-api-access-rszkv\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:19.073115 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.073111    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:19.073115 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.073122    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f913a96-21ec-4d86-b96a-1eea7c86da25-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:19.073391 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.073132    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f913a96-21ec-4d86-b96a-1eea7c86da25-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:19.664988 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.664949    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl" podUID="d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" containerName="main" containerID="cri-o://a6970fe23d13646465faaa9f2d360876e731352f6c64fdbc61179d60dd05d3b0" gracePeriod=45
Sep 11 18:24:19.841468 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.841436    2571 generic.go:356] "Generic (PLEG): container finished" podID="d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" containerID="a6970fe23d13646465faaa9f2d360876e731352f6c64fdbc61179d60dd05d3b0" exitCode=0
Sep 11 18:24:19.841638 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.841510    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl" event={"ID":"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032","Type":"ContainerDied","Data":"a6970fe23d13646465faaa9f2d360876e731352f6c64fdbc61179d60dd05d3b0"}
Sep 11 18:24:19.842983 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.842964    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"
Sep 11 18:24:19.843105 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.842964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg" event={"ID":"8f913a96-21ec-4d86-b96a-1eea7c86da25","Type":"ContainerDied","Data":"033b2e45269d9ca238601949f4e93e7fa0aca583b9c6c216b5ed0273df03f146"}
Sep 11 18:24:19.843105 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.843082    2571 scope.go:117] "RemoveContainer" containerID="4250b4a1ffb1aa789342e48ece620a9dbc66fee31978db7b1581cc0a01cb8652"
Sep 11 18:24:19.871262 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.871232    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"]
Sep 11 18:24:19.875887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.875860    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/sub-test-model-a5bbf6-kserve-6d67ddc947-fq8mg"]
Sep 11 18:24:19.913734 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.913713    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:24:19.982088 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.981995    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-model-cache\") pod \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") "
Sep 11 18:24:19.982088 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982054    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-dshm\") pod \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") "
Sep 11 18:24:19.982088 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982078    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tls-certs\") pod \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") "
Sep 11 18:24:19.982399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982117    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-home\") pod \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") "
Sep 11 18:24:19.982399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982141    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tmp-dir\") pod \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") "
Sep 11 18:24:19.982399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982236    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr7nz\" (UniqueName: \"kubernetes.io/projected/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-kube-api-access-rr7nz\") pod \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\" (UID: \"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032\") "
Sep 11 18:24:19.982399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982327    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-model-cache" (OuterVolumeSpecName: "model-cache") pod "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" (UID: "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:19.982591 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982406    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-home" (OuterVolumeSpecName: "home") pod "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" (UID: "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:19.982591 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982459    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" (UID: "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:19.982591 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982526    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:19.982591 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982539    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:19.982591 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.982553    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:19.984360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.984337    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-dshm" (OuterVolumeSpecName: "dshm") pod "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" (UID: "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:19.984450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.984434    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-kube-api-access-rr7nz" (OuterVolumeSpecName: "kube-api-access-rr7nz") pod "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" (UID: "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032"). InnerVolumeSpecName "kube-api-access-rr7nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:19.984522 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:19.984504    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" (UID: "d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:24:20.083143 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.083106    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7nz\" (UniqueName: \"kubernetes.io/projected/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-kube-api-access-rr7nz\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:20.083143 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.083136    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:20.083143 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.083146    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:24:20.731413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.731381    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f913a96-21ec-4d86-b96a-1eea7c86da25" path="/var/lib/kubelet/pods/8f913a96-21ec-4d86-b96a-1eea7c86da25/volumes"
Sep 11 18:24:20.850175 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.850145    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"
Sep 11 18:24:20.850383 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.850145    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl" event={"ID":"d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032","Type":"ContainerDied","Data":"5a8c724fde4113ec541c2017d99226cb65fd96026473061d73f1ce36b6af6ff9"}
Sep 11 18:24:20.850383 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.850287    2571 scope.go:117] "RemoveContainer" containerID="a6970fe23d13646465faaa9f2d360876e731352f6c64fdbc61179d60dd05d3b0"
Sep 11 18:24:20.868018 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.867993    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"]
Sep 11 18:24:20.872069 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:20.872048    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/sub-test-model-5d35e7-kserve-d78b764b5-qsrvl"]
Sep 11 18:24:22.324695 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.324654    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"]
Sep 11 18:24:22.325103 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.325088    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" containerName="main"
Sep 11 18:24:22.325164 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.325105    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" containerName="main"
Sep 11 18:24:22.325164 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.325123    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f913a96-21ec-4d86-b96a-1eea7c86da25" containerName="main"
Sep 11 18:24:22.325164 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.325129    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f913a96-21ec-4d86-b96a-1eea7c86da25" containerName="main"
Sep 11 18:24:22.325275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.325194    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f913a96-21ec-4d86-b96a-1eea7c86da25" containerName="main"
Sep 11 18:24:22.325275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.325206    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" containerName="main"
Sep 11 18:24:22.329804 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.329779    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.334441 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.334416    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"openshift-service-ca.crt\""
Sep 11 18:24:22.334575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.334465    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"rate-test-model-a5bbf6-kserve-self-signed-certs\""
Sep 11 18:24:22.334575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.334509    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"kube-root-ca.crt\""
Sep 11 18:24:22.334575 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.334465    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a5bbf6\"/\"default-dockercfg-dpfk7\""
Sep 11 18:24:22.339557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.339534    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"]
Sep 11 18:24:22.404023 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.403988    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-model-cache\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.404220 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.404048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-tmp-dir\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.404220 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.404117    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9faa99cd-77b7-468b-8e66-e10b9efec81f-tls-certs\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.404220 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.404152    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-home\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.404220 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.404175    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-dshm\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.404220 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.404192    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlsn\" (UniqueName: \"kubernetes.io/projected/9faa99cd-77b7-468b-8e66-e10b9efec81f-kube-api-access-4rlsn\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505351 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505276    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-home\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505359    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-dshm\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlsn\" (UniqueName: \"kubernetes.io/projected/9faa99cd-77b7-468b-8e66-e10b9efec81f-kube-api-access-4rlsn\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505434    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-model-cache\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505504    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-tmp-dir\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505524    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9faa99cd-77b7-468b-8e66-e10b9efec81f-tls-certs\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505833 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505680    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-home\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505833 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505804    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-model-cache\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.505945 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.505852    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-tmp-dir\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.507538 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.507518    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-dshm\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.507838 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.507800    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9faa99cd-77b7-468b-8e66-e10b9efec81f-tls-certs\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.513761 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.513739    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlsn\" (UniqueName: \"kubernetes.io/projected/9faa99cd-77b7-468b-8e66-e10b9efec81f-kube-api-access-4rlsn\") pod \"rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") " pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.641196 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.641100    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:22.732102 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.732067    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032" path="/var/lib/kubelet/pods/d249de2a-c06d-4a4b-b3fe-b7cbd5bdc032/volumes"
Sep 11 18:24:22.915563 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:22.915536    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"]
Sep 11 18:24:22.916863 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:24:22.916843    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9faa99cd_77b7_468b_8e66_e10b9efec81f.slice/crio-ca519455e0bd4eecec8812cec7a7364bd28e2ee78b60ff39a07fa235287df532 WatchSource:0}: Error finding container ca519455e0bd4eecec8812cec7a7364bd28e2ee78b60ff39a07fa235287df532: Status 404 returned error can't find the container with id ca519455e0bd4eecec8812cec7a7364bd28e2ee78b60ff39a07fa235287df532
Sep 11 18:24:23.876358 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:23.876323    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6" event={"ID":"9faa99cd-77b7-468b-8e66-e10b9efec81f","Type":"ContainerStarted","Data":"1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b"}
Sep 11 18:24:23.876358 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:23.876369    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6" event={"ID":"9faa99cd-77b7-468b-8e66-e10b9efec81f","Type":"ContainerStarted","Data":"ca519455e0bd4eecec8812cec7a7364bd28e2ee78b60ff39a07fa235287df532"}
Sep 11 18:24:23.894041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:23.893993    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6" podStartSLOduration=1.893978879 podStartE2EDuration="1.893978879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:23.893347708 +0000 UTC m=+1827.663414057" watchObservedRunningTime="2026-09-11 18:24:23.893978879 +0000 UTC m=+1827.664045228"
Sep 11 18:24:32.641348 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:32.641286    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:32.641911 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:32.641363    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:32.654522 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:32.654497    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:32.940736 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:32.940709    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:24:58.719106 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.719069    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"]
Sep 11 18:24:58.724195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.724165    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:58.726760 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.726730    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"rate-test-model-5d35e7-kserve-self-signed-certs\""
Sep 11 18:24:58.726884 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.726803    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"default-dockercfg-bktdb\""
Sep 11 18:24:58.726884 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.726833    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"kube-root-ca.crt\""
Sep 11 18:24:58.727936 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.727918    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"openshift-service-ca.crt\""
Sep 11 18:24:58.734087 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.734062    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"]
Sep 11 18:24:58.920629 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.920590    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-model-cache\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:58.920810 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.920641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-dshm\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:58.920810 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.920731    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp79t\" (UniqueName: \"kubernetes.io/projected/07962b59-b90f-486d-b411-f80e6e958b3b-kube-api-access-vp79t\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:58.920810 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.920765    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-home\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:58.920946 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.920840    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07962b59-b90f-486d-b411-f80e6e958b3b-tls-certs\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:58.920988 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:58.920952    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-tmp-dir\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022113 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022031    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-dshm\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022113 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022094    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp79t\" (UniqueName: \"kubernetes.io/projected/07962b59-b90f-486d-b411-f80e6e958b3b-kube-api-access-vp79t\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022113 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022112    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-home\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022418 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022139    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07962b59-b90f-486d-b411-f80e6e958b3b-tls-certs\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022418 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022157    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-tmp-dir\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022418 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-model-cache\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022573 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-home\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022632 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022578    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-tmp-dir\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.022632 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.022610    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-model-cache\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.024398 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.024377    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-dshm\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.024640 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.024624    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07962b59-b90f-486d-b411-f80e6e958b3b-tls-certs\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.030720 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.030699    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp79t\" (UniqueName: \"kubernetes.io/projected/07962b59-b90f-486d-b411-f80e6e958b3b-kube-api-access-vp79t\") pod \"rate-test-model-5d35e7-kserve-6db45d968c-t46sw\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.036509 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.036488    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:24:59.312707 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:24:59.312678    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"]
Sep 11 18:24:59.313653 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:24:59.313627    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07962b59_b90f_486d_b411_f80e6e958b3b.slice/crio-56e5d855d65fd0027835421c6436799890dcda91b3a43c7d675bc1aa6c62a59d WatchSource:0}: Error finding container 56e5d855d65fd0027835421c6436799890dcda91b3a43c7d675bc1aa6c62a59d: Status 404 returned error can't find the container with id 56e5d855d65fd0027835421c6436799890dcda91b3a43c7d675bc1aa6c62a59d
Sep 11 18:25:00.091560 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:00.091524    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw" event={"ID":"07962b59-b90f-486d-b411-f80e6e958b3b","Type":"ContainerStarted","Data":"e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd"}
Sep 11 18:25:00.091946 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:00.091568    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw" event={"ID":"07962b59-b90f-486d-b411-f80e6e958b3b","Type":"ContainerStarted","Data":"56e5d855d65fd0027835421c6436799890dcda91b3a43c7d675bc1aa6c62a59d"}
Sep 11 18:25:00.111033 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:00.110983    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw" podStartSLOduration=2.110967876 podStartE2EDuration="2.110967876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:00.109594263 +0000 UTC m=+1863.879660614" watchObservedRunningTime="2026-09-11 18:25:00.110967876 +0000 UTC m=+1863.881034267"
Sep 11 18:25:09.037136 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:09.037092    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:25:09.037656 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:09.037159    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:25:09.050710 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:09.050680    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:25:09.157769 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:09.157739    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:25:33.788007 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:33.787917    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"]
Sep 11 18:25:34.817563 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:34.817531    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"]
Sep 11 18:25:41.345059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.345022    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"]
Sep 11 18:25:41.348922 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.348896    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:41.351689 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.351661    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-be394b-dockercfg-x5hhm\""
Sep 11 18:25:41.351865 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.351668    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-be394b\""
Sep 11 18:25:41.359506 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.359480    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"]
Sep 11 18:25:41.505722 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.505689    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9pcj\" (UniqueName: \"kubernetes.io/projected/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-kube-api-access-s9pcj\") pod \"payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5\" (UID: \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:41.505915 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.505757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5\" (UID: \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:41.606906 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.606816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9pcj\" (UniqueName: \"kubernetes.io/projected/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-kube-api-access-s9pcj\") pod \"payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5\" (UID: \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:41.606906 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.606886    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5\" (UID: \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:41.607628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.607610    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5\" (UID: \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:41.615692 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.615671    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9pcj\" (UniqueName: \"kubernetes.io/projected/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-kube-api-access-s9pcj\") pod \"payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5\" (UID: \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:41.663743 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.663716    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:41.935511 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:41.935488    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"]
Sep 11 18:25:41.936965 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:25:41.936933    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24d3d9e_b330_4e21_b19b_7a3428e0b8d0.slice/crio-fef0208ea8430084ede6981ab720f947834efbf6227e13b5750d53cd3730ac40 WatchSource:0}: Error finding container fef0208ea8430084ede6981ab720f947834efbf6227e13b5750d53cd3730ac40: Status 404 returned error can't find the container with id fef0208ea8430084ede6981ab720f947834efbf6227e13b5750d53cd3730ac40
Sep 11 18:25:42.365789 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:42.365697    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5" event={"ID":"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0","Type":"ContainerStarted","Data":"75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f"}
Sep 11 18:25:42.366217 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:42.365796    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:25:42.366217 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:42.365814    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5" event={"ID":"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0","Type":"ContainerStarted","Data":"fef0208ea8430084ede6981ab720f947834efbf6227e13b5750d53cd3730ac40"}
Sep 11 18:25:42.384653 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:42.384607    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5" podStartSLOduration=1.384593027 podStartE2EDuration="1.384593027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:42.383509962 +0000 UTC m=+1906.153576310" watchObservedRunningTime="2026-09-11 18:25:42.384593027 +0000 UTC m=+1906.154659375"
Sep 11 18:25:48.808240 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:48.808177    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6" podUID="9faa99cd-77b7-468b-8e66-e10b9efec81f" containerName="main" containerID="cri-o://1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b" gracePeriod=45
Sep 11 18:25:49.056026 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.056003    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:25:49.070416 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070347    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-model-cache\") pod \"9faa99cd-77b7-468b-8e66-e10b9efec81f\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") "
Sep 11 18:25:49.070416 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070396    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rlsn\" (UniqueName: \"kubernetes.io/projected/9faa99cd-77b7-468b-8e66-e10b9efec81f-kube-api-access-4rlsn\") pod \"9faa99cd-77b7-468b-8e66-e10b9efec81f\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") "
Sep 11 18:25:49.070606 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070432    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-dshm\") pod \"9faa99cd-77b7-468b-8e66-e10b9efec81f\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") "
Sep 11 18:25:49.070606 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070462    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9faa99cd-77b7-468b-8e66-e10b9efec81f-tls-certs\") pod \"9faa99cd-77b7-468b-8e66-e10b9efec81f\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") "
Sep 11 18:25:49.070606 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070495    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-tmp-dir\") pod \"9faa99cd-77b7-468b-8e66-e10b9efec81f\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") "
Sep 11 18:25:49.070606 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070519    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-home\") pod \"9faa99cd-77b7-468b-8e66-e10b9efec81f\" (UID: \"9faa99cd-77b7-468b-8e66-e10b9efec81f\") "
Sep 11 18:25:49.070606 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070592    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9faa99cd-77b7-468b-8e66-e10b9efec81f" (UID: "9faa99cd-77b7-468b-8e66-e10b9efec81f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:49.070895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070751    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-home" (OuterVolumeSpecName: "home") pod "9faa99cd-77b7-468b-8e66-e10b9efec81f" (UID: "9faa99cd-77b7-468b-8e66-e10b9efec81f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:49.070895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070772    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:49.070895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.070777    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9faa99cd-77b7-468b-8e66-e10b9efec81f" (UID: "9faa99cd-77b7-468b-8e66-e10b9efec81f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:49.072769 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.072743    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-dshm" (OuterVolumeSpecName: "dshm") pod "9faa99cd-77b7-468b-8e66-e10b9efec81f" (UID: "9faa99cd-77b7-468b-8e66-e10b9efec81f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:49.072883 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.072822    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9faa99cd-77b7-468b-8e66-e10b9efec81f-kube-api-access-4rlsn" (OuterVolumeSpecName: "kube-api-access-4rlsn") pod "9faa99cd-77b7-468b-8e66-e10b9efec81f" (UID: "9faa99cd-77b7-468b-8e66-e10b9efec81f"). InnerVolumeSpecName "kube-api-access-4rlsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:25:49.072938 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.072893    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9faa99cd-77b7-468b-8e66-e10b9efec81f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9faa99cd-77b7-468b-8e66-e10b9efec81f" (UID: "9faa99cd-77b7-468b-8e66-e10b9efec81f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:25:49.172150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.172103    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:49.172150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.172160    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:49.172510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.172179    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rlsn\" (UniqueName: \"kubernetes.io/projected/9faa99cd-77b7-468b-8e66-e10b9efec81f-kube-api-access-4rlsn\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:49.172510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.172194    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9faa99cd-77b7-468b-8e66-e10b9efec81f-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:49.172510 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.172228    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9faa99cd-77b7-468b-8e66-e10b9efec81f-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:49.414499 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.414471    2571 generic.go:356] "Generic (PLEG): container finished" podID="9faa99cd-77b7-468b-8e66-e10b9efec81f" containerID="1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b" exitCode=0
Sep 11 18:25:49.414652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.414508    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"
Sep 11 18:25:49.414652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.414555    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6" event={"ID":"9faa99cd-77b7-468b-8e66-e10b9efec81f","Type":"ContainerDied","Data":"1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b"}
Sep 11 18:25:49.414652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.414595    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6" event={"ID":"9faa99cd-77b7-468b-8e66-e10b9efec81f","Type":"ContainerDied","Data":"ca519455e0bd4eecec8812cec7a7364bd28e2ee78b60ff39a07fa235287df532"}
Sep 11 18:25:49.414652 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.414615    2571 scope.go:117] "RemoveContainer" containerID="1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b"
Sep 11 18:25:49.424361 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.424344    2571 scope.go:117] "RemoveContainer" containerID="1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b"
Sep 11 18:25:49.424642 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:25:49.424625    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b\": container with ID starting with 1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b not found: ID does not exist" containerID="1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b"
Sep 11 18:25:49.424697 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.424650    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b"} err="failed to get container status \"1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b\": rpc error: code = NotFound desc = could not find container \"1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b\": container with ID starting with 1491bbb5b670e82b19265612f577b3538da61fcebb22c1a1ad934682bde9ed7b not found: ID does not exist"
Sep 11 18:25:49.436699 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.436676    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"]
Sep 11 18:25:49.439772 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.439751    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a5bbf6/rate-test-model-a5bbf6-kserve-68f79c864c-k4lj6"]
Sep 11 18:25:49.836490 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:49.836414    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw" podUID="07962b59-b90f-486d-b411-f80e6e958b3b" containerName="main" containerID="cri-o://e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd" gracePeriod=45
Sep 11 18:25:50.091009 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.090956    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:25:50.180033 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180003    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-home\") pod \"07962b59-b90f-486d-b411-f80e6e958b3b\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") "
Sep 11 18:25:50.180203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180042    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-model-cache\") pod \"07962b59-b90f-486d-b411-f80e6e958b3b\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") "
Sep 11 18:25:50.180203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180062    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-dshm\") pod \"07962b59-b90f-486d-b411-f80e6e958b3b\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") "
Sep 11 18:25:50.180203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180093    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07962b59-b90f-486d-b411-f80e6e958b3b-tls-certs\") pod \"07962b59-b90f-486d-b411-f80e6e958b3b\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") "
Sep 11 18:25:50.180203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180123    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-tmp-dir\") pod \"07962b59-b90f-486d-b411-f80e6e958b3b\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") "
Sep 11 18:25:50.180203 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180181    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp79t\" (UniqueName: \"kubernetes.io/projected/07962b59-b90f-486d-b411-f80e6e958b3b-kube-api-access-vp79t\") pod \"07962b59-b90f-486d-b411-f80e6e958b3b\" (UID: \"07962b59-b90f-486d-b411-f80e6e958b3b\") "
Sep 11 18:25:50.180518 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180289    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-home" (OuterVolumeSpecName: "home") pod "07962b59-b90f-486d-b411-f80e6e958b3b" (UID: "07962b59-b90f-486d-b411-f80e6e958b3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:50.180518 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180331    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "07962b59-b90f-486d-b411-f80e6e958b3b" (UID: "07962b59-b90f-486d-b411-f80e6e958b3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:50.180518 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180425    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07962b59-b90f-486d-b411-f80e6e958b3b" (UID: "07962b59-b90f-486d-b411-f80e6e958b3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:50.180518 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180498    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:50.180518 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.180516    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:50.182191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.182165    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-dshm" (OuterVolumeSpecName: "dshm") pod "07962b59-b90f-486d-b411-f80e6e958b3b" (UID: "07962b59-b90f-486d-b411-f80e6e958b3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:50.182321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.182193    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07962b59-b90f-486d-b411-f80e6e958b3b-kube-api-access-vp79t" (OuterVolumeSpecName: "kube-api-access-vp79t") pod "07962b59-b90f-486d-b411-f80e6e958b3b" (UID: "07962b59-b90f-486d-b411-f80e6e958b3b"). InnerVolumeSpecName "kube-api-access-vp79t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:25:50.182321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.182212    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07962b59-b90f-486d-b411-f80e6e958b3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07962b59-b90f-486d-b411-f80e6e958b3b" (UID: "07962b59-b90f-486d-b411-f80e6e958b3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:25:50.281123 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.281093    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:50.281123 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.281119    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp79t\" (UniqueName: \"kubernetes.io/projected/07962b59-b90f-486d-b411-f80e6e958b3b-kube-api-access-vp79t\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:50.281334 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.281142    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07962b59-b90f-486d-b411-f80e6e958b3b-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:50.281334 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.281151    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07962b59-b90f-486d-b411-f80e6e958b3b-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:25:50.422684 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.422655    2571 generic.go:356] "Generic (PLEG): container finished" podID="07962b59-b90f-486d-b411-f80e6e958b3b" containerID="e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd" exitCode=0
Sep 11 18:25:50.422872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.422695    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"
Sep 11 18:25:50.422872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.422740    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw" event={"ID":"07962b59-b90f-486d-b411-f80e6e958b3b","Type":"ContainerDied","Data":"e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd"}
Sep 11 18:25:50.422872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.422779    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw" event={"ID":"07962b59-b90f-486d-b411-f80e6e958b3b","Type":"ContainerDied","Data":"56e5d855d65fd0027835421c6436799890dcda91b3a43c7d675bc1aa6c62a59d"}
Sep 11 18:25:50.422872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.422796    2571 scope.go:117] "RemoveContainer" containerID="e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd"
Sep 11 18:25:50.432111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.432094    2571 scope.go:117] "RemoveContainer" containerID="e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd"
Sep 11 18:25:50.432397 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:25:50.432379    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd\": container with ID starting with e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd not found: ID does not exist" containerID="e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd"
Sep 11 18:25:50.432471 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.432404    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd"} err="failed to get container status \"e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd\": rpc error: code = NotFound desc = could not find container \"e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd\": container with ID starting with e2e965c337edee530a0a0b560bb12cd1a7472582d6928976e988a0467fd281dd not found: ID does not exist"
Sep 11 18:25:50.444204 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.444182    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"]
Sep 11 18:25:50.447924 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.447904    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/rate-test-model-5d35e7-kserve-6db45d968c-t46sw"]
Sep 11 18:25:50.730871 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.730795    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07962b59-b90f-486d-b411-f80e6e958b3b" path="/var/lib/kubelet/pods/07962b59-b90f-486d-b411-f80e6e958b3b/volumes"
Sep 11 18:25:50.731224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:50.731208    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9faa99cd-77b7-468b-8e66-e10b9efec81f" path="/var/lib/kubelet/pods/9faa99cd-77b7-468b-8e66-e10b9efec81f/volumes"
Sep 11 18:25:53.375057 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:25:53.375029    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:26:04.846896 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.846861    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"]
Sep 11 18:26:04.847286 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.847273    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07962b59-b90f-486d-b411-f80e6e958b3b" containerName="main"
Sep 11 18:26:04.847390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.847287    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="07962b59-b90f-486d-b411-f80e6e958b3b" containerName="main"
Sep 11 18:26:04.847390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.847317    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9faa99cd-77b7-468b-8e66-e10b9efec81f" containerName="main"
Sep 11 18:26:04.847390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.847323    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9faa99cd-77b7-468b-8e66-e10b9efec81f" containerName="main"
Sep 11 18:26:04.847545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.847419    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9faa99cd-77b7-468b-8e66-e10b9efec81f" containerName="main"
Sep 11 18:26:04.847545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.847433    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="07962b59-b90f-486d-b411-f80e6e958b3b" containerName="main"
Sep 11 18:26:04.850725 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.850701    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.853489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.853463    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-74a302-openshift-default-dockercfg-z8qmh\""
Sep 11 18:26:04.866775 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.866751    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"]
Sep 11 18:26:04.912217 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912190    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-envoy\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.912399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912225    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-certs\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.912399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912252    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istiod-ca-cert\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.912399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912340    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-token\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.912399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912380    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-socket\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.912562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-credential-socket\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.912562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912450    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-data\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.912562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcc6\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-kube-api-access-gvcc6\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:04.912562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:04.912502    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-podinfo\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013716 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013680    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-socket\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013737    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-credential-socket\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-data\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013778    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcc6\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-kube-api-access-gvcc6\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-podinfo\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013825    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-envoy\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013852    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-certs\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013890    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istiod-ca-cert\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.013972 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.013930    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-token\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.014403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.014141    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-data\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.014403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.014162    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-socket\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.014403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.014218    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-credential-socket\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.014610 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.014576    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-certs\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.014732 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.014715    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istiod-ca-cert\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.016329 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.016291    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-envoy\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.016413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.016343    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-podinfo\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.023185 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.023143    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-token\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.023608 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.023580    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcc6\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-kube-api-access-gvcc6\") pod \"e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") " pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.163860 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.163826    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:05.439565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.439534    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"]
Sep 11 18:26:05.441668 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:26:05.441643    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2693f0ee_2647_4c60_b4f2_5937f1e856a0.slice/crio-83d3bc9c23583ca3d96973eec0b29ded13f124a261eb2958a686f2f224f8eb04 WatchSource:0}: Error finding container 83d3bc9c23583ca3d96973eec0b29ded13f124a261eb2958a686f2f224f8eb04: Status 404 returned error can't find the container with id 83d3bc9c23583ca3d96973eec0b29ded13f124a261eb2958a686f2f224f8eb04
Sep 11 18:26:05.444065 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.444028    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:26:05.444141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.444113    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:26:05.444185 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.444156    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:26:05.511500 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:05.511466    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" event={"ID":"2693f0ee-2647-4c60-b4f2-5937f1e856a0","Type":"ContainerStarted","Data":"83d3bc9c23583ca3d96973eec0b29ded13f124a261eb2958a686f2f224f8eb04"}
Sep 11 18:26:06.518949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:06.518917    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" event={"ID":"2693f0ee-2647-4c60-b4f2-5937f1e856a0","Type":"ContainerStarted","Data":"74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5"}
Sep 11 18:26:06.543585 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:06.543537    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" podStartSLOduration=2.543520907 podStartE2EDuration="2.543520907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:06.540985662 +0000 UTC m=+1930.311052032" watchObservedRunningTime="2026-09-11 18:26:06.543520907 +0000 UTC m=+1930.313587254"
Sep 11 18:26:07.165036 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:07.164994    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:07.170003 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:07.169974    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:07.525847 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:07.525765    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:07.526662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:07.526643    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:11.270096 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.270061    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"]
Sep 11 18:26:11.273693 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.273674    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:11.276490 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.276470    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-74a302\""
Sep 11 18:26:11.276630 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.276510    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-74a302-dockercfg-5lbdn\""
Sep 11 18:26:11.283940 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.283919    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"]
Sep 11 18:26:11.292107 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.292082    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"]
Sep 11 18:26:11.296349 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.296330    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:11.302316 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.302283    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"]
Sep 11 18:26:11.372210 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.372175    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j4xn\" (UniqueName: \"kubernetes.io/projected/a2c08d5f-9e82-476d-8c9a-763f35087537-kube-api-access-2j4xn\") pod \"payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl\" (UID: \"a2c08d5f-9e82-476d-8c9a-763f35087537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:11.372444 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.372219    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b513bd-d35f-478f-aa45-69a05b126219-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz\" (UID: \"27b513bd-d35f-478f-aa45-69a05b126219\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:11.372444 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.372419    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sfvd\" (UniqueName: \"kubernetes.io/projected/27b513bd-d35f-478f-aa45-69a05b126219-kube-api-access-7sfvd\") pod \"payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz\" (UID: \"27b513bd-d35f-478f-aa45-69a05b126219\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:11.372545 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.372460    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c08d5f-9e82-476d-8c9a-763f35087537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl\" (UID: \"a2c08d5f-9e82-476d-8c9a-763f35087537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:11.473101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.473067    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sfvd\" (UniqueName: \"kubernetes.io/projected/27b513bd-d35f-478f-aa45-69a05b126219-kube-api-access-7sfvd\") pod \"payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz\" (UID: \"27b513bd-d35f-478f-aa45-69a05b126219\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:11.473101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.473110    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c08d5f-9e82-476d-8c9a-763f35087537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl\" (UID: \"a2c08d5f-9e82-476d-8c9a-763f35087537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:11.473390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.473184    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j4xn\" (UniqueName: \"kubernetes.io/projected/a2c08d5f-9e82-476d-8c9a-763f35087537-kube-api-access-2j4xn\") pod \"payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl\" (UID: \"a2c08d5f-9e82-476d-8c9a-763f35087537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:11.473390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.473205    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b513bd-d35f-478f-aa45-69a05b126219-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz\" (UID: \"27b513bd-d35f-478f-aa45-69a05b126219\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:11.473959 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.473933    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c08d5f-9e82-476d-8c9a-763f35087537-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl\" (UID: \"a2c08d5f-9e82-476d-8c9a-763f35087537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:11.474047 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.473960    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b513bd-d35f-478f-aa45-69a05b126219-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz\" (UID: \"27b513bd-d35f-478f-aa45-69a05b126219\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:11.481940 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.481916    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sfvd\" (UniqueName: \"kubernetes.io/projected/27b513bd-d35f-478f-aa45-69a05b126219-kube-api-access-7sfvd\") pod \"payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz\" (UID: \"27b513bd-d35f-478f-aa45-69a05b126219\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:11.482025 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.481939    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j4xn\" (UniqueName: \"kubernetes.io/projected/a2c08d5f-9e82-476d-8c9a-763f35087537-kube-api-access-2j4xn\") pod \"payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl\" (UID: \"a2c08d5f-9e82-476d-8c9a-763f35087537\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:11.585283 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.585202    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:11.607042 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.607015    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:11.860767 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.860738    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"]
Sep 11 18:26:11.861781 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:26:11.861756    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c08d5f_9e82_476d_8c9a_763f35087537.slice/crio-b8356d2397c0490571a896ea7af7b08e4b1b495d2005cee77d43ed417b67f8df WatchSource:0}: Error finding container b8356d2397c0490571a896ea7af7b08e4b1b495d2005cee77d43ed417b67f8df: Status 404 returned error can't find the container with id b8356d2397c0490571a896ea7af7b08e4b1b495d2005cee77d43ed417b67f8df
Sep 11 18:26:11.894191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:11.894161    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"]
Sep 11 18:26:11.902483 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:26:11.902456    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b513bd_d35f_478f_aa45_69a05b126219.slice/crio-35174de41ba06088f8680a6ae7bba4a6e423805de0200bf977b72f90408c685e WatchSource:0}: Error finding container 35174de41ba06088f8680a6ae7bba4a6e423805de0200bf977b72f90408c685e: Status 404 returned error can't find the container with id 35174de41ba06088f8680a6ae7bba4a6e423805de0200bf977b72f90408c685e
Sep 11 18:26:12.557504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:12.557465    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz" event={"ID":"27b513bd-d35f-478f-aa45-69a05b126219","Type":"ContainerStarted","Data":"584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497"}
Sep 11 18:26:12.557993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:12.557532    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz" event={"ID":"27b513bd-d35f-478f-aa45-69a05b126219","Type":"ContainerStarted","Data":"35174de41ba06088f8680a6ae7bba4a6e423805de0200bf977b72f90408c685e"}
Sep 11 18:26:12.557993 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:12.557672    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:12.559277 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:12.559251    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl" event={"ID":"a2c08d5f-9e82-476d-8c9a-763f35087537","Type":"ContainerStarted","Data":"ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151"}
Sep 11 18:26:12.559407 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:12.559320    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:12.559407 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:12.559341    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl" event={"ID":"a2c08d5f-9e82-476d-8c9a-763f35087537","Type":"ContainerStarted","Data":"b8356d2397c0490571a896ea7af7b08e4b1b495d2005cee77d43ed417b67f8df"}
Sep 11 18:26:12.576414 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:12.576363    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz" podStartSLOduration=1.576344545 podStartE2EDuration="1.576344545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:12.574524306 +0000 UTC m=+1936.344590655" watchObservedRunningTime="2026-09-11 18:26:12.576344545 +0000 UTC m=+1936.346410897"
Sep 11 18:26:12.590955 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:12.590912    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl" podStartSLOduration=1.5908942430000002 podStartE2EDuration="1.590894243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:12.590715836 +0000 UTC m=+1936.360782184" watchObservedRunningTime="2026-09-11 18:26:12.590894243 +0000 UTC m=+1936.360960628"
Sep 11 18:26:23.568350 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:23.568313    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:26:23.568839 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:23.568499    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:26:26.651502 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:26.651466    2571 generic.go:356] "Generic (PLEG): container finished" podID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerID="74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5" exitCode=0
Sep 11 18:26:26.652036 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:26.651542    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" event={"ID":"2693f0ee-2647-4c60-b4f2-5937f1e856a0","Type":"ContainerDied","Data":"74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5"}
Sep 11 18:26:26.652036 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:26.652010    2571 scope.go:117] "RemoveContainer" containerID="74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5"
Sep 11 18:26:26.653139 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:26.653104    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:26:26.653248 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:26.653192    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:26:26.653248 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:26.653240    2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:26:27.670080 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:27.670040    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" event={"ID":"2693f0ee-2647-4c60-b4f2-5937f1e856a0","Type":"ContainerStarted","Data":"e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a"}
Sep 11 18:26:28.165227 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:28.165188    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:28.165463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:28.165342    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Sep 11 18:26:28.165463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:28.165390    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Sep 11 18:26:29.165582 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:29.165051    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Sep 11 18:26:29.165582 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:29.165141    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Sep 11 18:26:30.165263 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:30.165223    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Sep 11 18:26:30.165614 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:30.165567    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Sep 11 18:26:31.166159 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:31.166125    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Sep 11 18:26:31.166783 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:31.166728    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Sep 11 18:26:32.164639 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:32.164595    2571 patch_prober.go:28] interesting pod/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Sep 11 18:26:32.164856 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:32.164676    2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Sep 11 18:26:33.168127 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:33.168095    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:33.168602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:33.168552    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:33.169360 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:33.169343    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:26:38.399794 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:38.399761    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g2ncf_34cfa5cb-dbc3-4c99-903e-944f864cef32/payload-processing/0.log"
Sep 11 18:26:38.517672 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:38.517644    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz_27b513bd-d35f-478f-aa45-69a05b126219/payload-processing/0.log"
Sep 11 18:26:57.418713 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:57.418639    2571 scope.go:117] "RemoveContainer" containerID="2b2e1f96d631c0231c04862ab95febb085aca830df51bf27b4e6cf20cb8c0942"
Sep 11 18:26:57.427235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:26:57.427214    2571 scope.go:117] "RemoveContainer" containerID="fd579d2ccb3b842d744dd5a6e12a381a3a4d56212c90f3776bc0e7d46812d01e"
Sep 11 18:27:09.240340 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:09.240288    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g2ncf_34cfa5cb-dbc3-4c99-903e-944f864cef32/payload-processing/0.log"
Sep 11 18:27:09.375753 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:09.375724    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz_27b513bd-d35f-478f-aa45-69a05b126219/payload-processing/0.log"
Sep 11 18:27:17.225322 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.225276    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"]
Sep 11 18:27:17.229059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.229035    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.232598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.232575    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-b2aace\""
Sep 11 18:27:17.241312 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.241267    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"]
Sep 11 18:27:17.245799 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.245777    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"]
Sep 11 18:27:17.249210 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.249191    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.253973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.253940    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-b2aace-dockercfg-4tnmj\""
Sep 11 18:27:17.264225 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.264202    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"]
Sep 11 18:27:17.374084 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.374048    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkf9b\" (UniqueName: \"kubernetes.io/projected/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-kube-api-access-bkf9b\") pod \"payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz\" (UID: \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.374273 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.374106    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42db42c7-1079-4b7c-85cb-3b38008da53c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc\" (UID: \"42db42c7-1079-4b7c-85cb-3b38008da53c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.374273 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.374242    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz\" (UID: \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.374392 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.374290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlq9t\" (UniqueName: \"kubernetes.io/projected/42db42c7-1079-4b7c-85cb-3b38008da53c-kube-api-access-hlq9t\") pod \"payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc\" (UID: \"42db42c7-1079-4b7c-85cb-3b38008da53c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.475174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.475135    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz\" (UID: \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.475174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.475178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlq9t\" (UniqueName: \"kubernetes.io/projected/42db42c7-1079-4b7c-85cb-3b38008da53c-kube-api-access-hlq9t\") pod \"payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc\" (UID: \"42db42c7-1079-4b7c-85cb-3b38008da53c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.475512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.475206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkf9b\" (UniqueName: \"kubernetes.io/projected/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-kube-api-access-bkf9b\") pod \"payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz\" (UID: \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.475512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.475231    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42db42c7-1079-4b7c-85cb-3b38008da53c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc\" (UID: \"42db42c7-1079-4b7c-85cb-3b38008da53c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.475942 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.475919    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42db42c7-1079-4b7c-85cb-3b38008da53c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc\" (UID: \"42db42c7-1079-4b7c-85cb-3b38008da53c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.476030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.475984    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz\" (UID: \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.483981 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.483957    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlq9t\" (UniqueName: \"kubernetes.io/projected/42db42c7-1079-4b7c-85cb-3b38008da53c-kube-api-access-hlq9t\") pod \"payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc\" (UID: \"42db42c7-1079-4b7c-85cb-3b38008da53c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.484178 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.484146    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkf9b\" (UniqueName: \"kubernetes.io/projected/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-kube-api-access-bkf9b\") pod \"payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz\" (UID: \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.539275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.539239    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.560847 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.560819    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.816320 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.816277    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"]
Sep 11 18:27:17.817262 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:27:17.817238    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837797d7_5c6e_4a09_8d0f_36bcfd1a0662.slice/crio-b0509c141530455ddd05d0a9363c748de86febe4cea1e9c9285a16a0552e41fd WatchSource:0}: Error finding container b0509c141530455ddd05d0a9363c748de86febe4cea1e9c9285a16a0552e41fd: Status 404 returned error can't find the container with id b0509c141530455ddd05d0a9363c748de86febe4cea1e9c9285a16a0552e41fd
Sep 11 18:27:17.844028 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.844004    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"]
Sep 11 18:27:17.850665 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:27:17.850638    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42db42c7_1079_4b7c_85cb_3b38008da53c.slice/crio-45ea5bb23427c4240b3df49ba8422115bf603e3d834fcdb6c54807b325daeb75 WatchSource:0}: Error finding container 45ea5bb23427c4240b3df49ba8422115bf603e3d834fcdb6c54807b325daeb75: Status 404 returned error can't find the container with id 45ea5bb23427c4240b3df49ba8422115bf603e3d834fcdb6c54807b325daeb75
Sep 11 18:27:17.971984 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.971926    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz" event={"ID":"837797d7-5c6e-4a09-8d0f-36bcfd1a0662","Type":"ContainerStarted","Data":"a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b"}
Sep 11 18:27:17.972239 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.972007    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:17.972239 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.972033    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz" event={"ID":"837797d7-5c6e-4a09-8d0f-36bcfd1a0662","Type":"ContainerStarted","Data":"b0509c141530455ddd05d0a9363c748de86febe4cea1e9c9285a16a0552e41fd"}
Sep 11 18:27:17.973715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.973690    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc" event={"ID":"42db42c7-1079-4b7c-85cb-3b38008da53c","Type":"ContainerStarted","Data":"9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b"}
Sep 11 18:27:17.973828 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.973734    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc" event={"ID":"42db42c7-1079-4b7c-85cb-3b38008da53c","Type":"ContainerStarted","Data":"45ea5bb23427c4240b3df49ba8422115bf603e3d834fcdb6c54807b325daeb75"}
Sep 11 18:27:17.973828 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.973758    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:17.992061 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:17.992015    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz" podStartSLOduration=0.991999453 podStartE2EDuration="991.999453ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:17.987928609 +0000 UTC m=+2001.757994957" watchObservedRunningTime="2026-09-11 18:27:17.991999453 +0000 UTC m=+2001.762065800"
Sep 11 18:27:18.007722 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:18.007672    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc" podStartSLOduration=1.007658489 podStartE2EDuration="1.007658489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:18.005230497 +0000 UTC m=+2001.775296849" watchObservedRunningTime="2026-09-11 18:27:18.007658489 +0000 UTC m=+2001.777724837"
Sep 11 18:27:28.983357 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:28.983319    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:28.983763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:28.983410    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:44.173620 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.173584    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"]
Sep 11 18:27:44.177069 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.177050    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"
Sep 11 18:27:44.180186 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.180166    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:27:44.188092 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.188069    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"]
Sep 11 18:27:44.203831 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.203808    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffc6d-9eae-4ab7-aa27-90e0abfc11af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd\" (UID: \"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"
Sep 11 18:27:44.305034 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.305007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffc6d-9eae-4ab7-aa27-90e0abfc11af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd\" (UID: \"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"
Sep 11 18:27:44.305612 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.305594    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffc6d-9eae-4ab7-aa27-90e0abfc11af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd\" (UID: \"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"
Sep 11 18:27:44.488621 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.488539    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"
Sep 11 18:27:44.756723 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:44.756696    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"]
Sep 11 18:27:44.757180 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:27:44.757155    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43ffc6d_9eae_4ab7_aa27_90e0abfc11af.slice/crio-b6459d04d1a935bf07d5fe45a3d7989a2a8f665895d0070d7006f9a3a765ca53 WatchSource:0}: Error finding container b6459d04d1a935bf07d5fe45a3d7989a2a8f665895d0070d7006f9a3a765ca53: Status 404 returned error can't find the container with id b6459d04d1a935bf07d5fe45a3d7989a2a8f665895d0070d7006f9a3a765ca53
Sep 11 18:27:45.145390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:45.145353    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd" event={"ID":"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af","Type":"ContainerStarted","Data":"dc9151f279dd6be068fbed09bbba5487e2024bfc5cebbfda22508cafec31d18a"}
Sep 11 18:27:45.145567 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:45.145407    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd" event={"ID":"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af","Type":"ContainerStarted","Data":"b6459d04d1a935bf07d5fe45a3d7989a2a8f665895d0070d7006f9a3a765ca53"}
Sep 11 18:27:45.163807 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:45.163755    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd" podStartSLOduration=1.163740639 podStartE2EDuration="1.163740639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:45.160753813 +0000 UTC m=+2028.930820184" watchObservedRunningTime="2026-09-11 18:27:45.163740639 +0000 UTC m=+2028.933806986"
Sep 11 18:27:46.152678 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:46.152644    2571 generic.go:356] "Generic (PLEG): container finished" podID="c43ffc6d-9eae-4ab7-aa27-90e0abfc11af" containerID="dc9151f279dd6be068fbed09bbba5487e2024bfc5cebbfda22508cafec31d18a" exitCode=0
Sep 11 18:27:46.153111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:46.152732    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd" event={"ID":"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af","Type":"ContainerDied","Data":"dc9151f279dd6be068fbed09bbba5487e2024bfc5cebbfda22508cafec31d18a"}
Sep 11 18:27:47.289314 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:47.289277    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"
Sep 11 18:27:47.332634 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:47.332609    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffc6d-9eae-4ab7-aa27-90e0abfc11af-maas-api-service-ca\") pod \"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af\" (UID: \"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af\") "
Sep 11 18:27:47.332983 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:47.332964    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43ffc6d-9eae-4ab7-aa27-90e0abfc11af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c43ffc6d-9eae-4ab7-aa27-90e0abfc11af" (UID: "c43ffc6d-9eae-4ab7-aa27-90e0abfc11af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:27:47.433714 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:47.433639    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffc6d-9eae-4ab7-aa27-90e0abfc11af-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:27:48.165818 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:48.165786    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd" event={"ID":"c43ffc6d-9eae-4ab7-aa27-90e0abfc11af","Type":"ContainerDied","Data":"b6459d04d1a935bf07d5fe45a3d7989a2a8f665895d0070d7006f9a3a765ca53"}
Sep 11 18:27:48.165818 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:48.165810    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"
Sep 11 18:27:48.166025 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:48.165834    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6459d04d1a935bf07d5fe45a3d7989a2a8f665895d0070d7006f9a3a765ca53"
Sep 11 18:27:55.477433 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.476789    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"]
Sep 11 18:27:55.481232 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.481197    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc" podUID="42db42c7-1079-4b7c-85cb-3b38008da53c" containerName="payload-processing" containerID="cri-o://9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b" gracePeriod=30
Sep 11 18:27:55.485860 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.485835    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"]
Sep 11 18:27:55.487947 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.487887    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz" podUID="837797d7-5c6e-4a09-8d0f-36bcfd1a0662" containerName="payload-pre-processing" containerID="cri-o://a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b" gracePeriod=30
Sep 11 18:27:55.633441 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.633390    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"]
Sep 11 18:27:55.635789 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.635756    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-b2aace-rvlfd"]
Sep 11 18:27:55.804827 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.804794    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:55.809782 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.809472    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:55.905879 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.905847    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlq9t\" (UniqueName: \"kubernetes.io/projected/42db42c7-1079-4b7c-85cb-3b38008da53c-kube-api-access-hlq9t\") pod \"42db42c7-1079-4b7c-85cb-3b38008da53c\" (UID: \"42db42c7-1079-4b7c-85cb-3b38008da53c\") "
Sep 11 18:27:55.906064 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.905885    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkf9b\" (UniqueName: \"kubernetes.io/projected/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-kube-api-access-bkf9b\") pod \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\" (UID: \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\") "
Sep 11 18:27:55.906064 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.905905    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42db42c7-1079-4b7c-85cb-3b38008da53c-plugins-config-volume\") pod \"42db42c7-1079-4b7c-85cb-3b38008da53c\" (UID: \"42db42c7-1079-4b7c-85cb-3b38008da53c\") "
Sep 11 18:27:55.906064 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.905956    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-plugins-config-volume\") pod \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\" (UID: \"837797d7-5c6e-4a09-8d0f-36bcfd1a0662\") "
Sep 11 18:27:55.906365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.906340    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42db42c7-1079-4b7c-85cb-3b38008da53c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42db42c7-1079-4b7c-85cb-3b38008da53c" (UID: "42db42c7-1079-4b7c-85cb-3b38008da53c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:27:55.906434 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.906349    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "837797d7-5c6e-4a09-8d0f-36bcfd1a0662" (UID: "837797d7-5c6e-4a09-8d0f-36bcfd1a0662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:27:55.907955 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.907933    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-kube-api-access-bkf9b" (OuterVolumeSpecName: "kube-api-access-bkf9b") pod "837797d7-5c6e-4a09-8d0f-36bcfd1a0662" (UID: "837797d7-5c6e-4a09-8d0f-36bcfd1a0662"). InnerVolumeSpecName "kube-api-access-bkf9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:55.908025 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:55.908000    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42db42c7-1079-4b7c-85cb-3b38008da53c-kube-api-access-hlq9t" (OuterVolumeSpecName: "kube-api-access-hlq9t") pod "42db42c7-1079-4b7c-85cb-3b38008da53c" (UID: "42db42c7-1079-4b7c-85cb-3b38008da53c"). InnerVolumeSpecName "kube-api-access-hlq9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:56.007339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.007280    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlq9t\" (UniqueName: \"kubernetes.io/projected/42db42c7-1079-4b7c-85cb-3b38008da53c-kube-api-access-hlq9t\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:27:56.007339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.007335    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkf9b\" (UniqueName: \"kubernetes.io/projected/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-kube-api-access-bkf9b\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:27:56.007339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.007347    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42db42c7-1079-4b7c-85cb-3b38008da53c-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:27:56.007565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.007358    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837797d7-5c6e-4a09-8d0f-36bcfd1a0662-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:27:56.222536 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.222502    2571 generic.go:356] "Generic (PLEG): container finished" podID="42db42c7-1079-4b7c-85cb-3b38008da53c" containerID="9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b" exitCode=0
Sep 11 18:27:56.222724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.222546    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"
Sep 11 18:27:56.222724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.222586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc" event={"ID":"42db42c7-1079-4b7c-85cb-3b38008da53c","Type":"ContainerDied","Data":"9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b"}
Sep 11 18:27:56.222724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.222643    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc" event={"ID":"42db42c7-1079-4b7c-85cb-3b38008da53c","Type":"ContainerDied","Data":"45ea5bb23427c4240b3df49ba8422115bf603e3d834fcdb6c54807b325daeb75"}
Sep 11 18:27:56.222724 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.222671    2571 scope.go:117] "RemoveContainer" containerID="9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b"
Sep 11 18:27:56.224077 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.224048    2571 generic.go:356] "Generic (PLEG): container finished" podID="837797d7-5c6e-4a09-8d0f-36bcfd1a0662" containerID="a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b" exitCode=0
Sep 11 18:27:56.224198 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.224086    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"
Sep 11 18:27:56.224198 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.224117    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz" event={"ID":"837797d7-5c6e-4a09-8d0f-36bcfd1a0662","Type":"ContainerDied","Data":"a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b"}
Sep 11 18:27:56.224198 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.224151    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz" event={"ID":"837797d7-5c6e-4a09-8d0f-36bcfd1a0662","Type":"ContainerDied","Data":"b0509c141530455ddd05d0a9363c748de86febe4cea1e9c9285a16a0552e41fd"}
Sep 11 18:27:56.232160 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.232143    2571 scope.go:117] "RemoveContainer" containerID="9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b"
Sep 11 18:27:56.232438 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:27:56.232420    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b\": container with ID starting with 9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b not found: ID does not exist" containerID="9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b"
Sep 11 18:27:56.232494 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.232444    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b"} err="failed to get container status \"9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b\": rpc error: code = NotFound desc = could not find container \"9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b\": container with ID starting with 9e83e8dd7df68de9492fb9fcb5ff94d5e4727ca4b63bc7fbe51405826a468f3b not found: ID does not exist"
Sep 11 18:27:56.232494 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.232459    2571 scope.go:117] "RemoveContainer" containerID="a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b"
Sep 11 18:27:56.240803 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.240787    2571 scope.go:117] "RemoveContainer" containerID="a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b"
Sep 11 18:27:56.241051 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:27:56.241035    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b\": container with ID starting with a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b not found: ID does not exist" containerID="a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b"
Sep 11 18:27:56.241091 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.241058    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b"} err="failed to get container status \"a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b\": rpc error: code = NotFound desc = could not find container \"a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b\": container with ID starting with a4cf4d2a1d3e5fa38072603120b52a8388b6d29c4b176e7bd14f6b91abda856b not found: ID does not exist"
Sep 11 18:27:56.245450 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.245431    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"]
Sep 11 18:27:56.249041 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.249020    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-b2aace-647c5b96f9-6fzrc"]
Sep 11 18:27:56.259105 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.259084    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"]
Sep 11 18:27:56.264815 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.264790    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-b2aace-5b4fbfb4b-lvvdz"]
Sep 11 18:27:56.731609 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.731577    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42db42c7-1079-4b7c-85cb-3b38008da53c" path="/var/lib/kubelet/pods/42db42c7-1079-4b7c-85cb-3b38008da53c/volumes"
Sep 11 18:27:56.732104 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.732086    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="837797d7-5c6e-4a09-8d0f-36bcfd1a0662" path="/var/lib/kubelet/pods/837797d7-5c6e-4a09-8d0f-36bcfd1a0662/volumes"
Sep 11 18:27:56.732464 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:56.732447    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c43ffc6d-9eae-4ab7-aa27-90e0abfc11af" path="/var/lib/kubelet/pods/c43ffc6d-9eae-4ab7-aa27-90e0abfc11af/volumes"
Sep 11 18:27:57.451607 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:57.451579    2571 scope.go:117] "RemoveContainer" containerID="c6a38a37b71384f94e9947cd9ec77b5fbd25529c7035d7b382343bef1ba1975d"
Sep 11 18:27:57.461654 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:57.461635    2571 scope.go:117] "RemoveContainer" containerID="f1d3ed1246e5bbda9eb062c38ef4fe606ddb14ab1f731b73d6a1404e78648b0f"
Sep 11 18:27:57.469893 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:27:57.469877    2571 scope.go:117] "RemoveContainer" containerID="387023819ef2c72ed2dbfa3b2caa18f7473d0279776cbb9e59a4321f71f6e65e"
Sep 11 18:28:06.081500 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.081463    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"]
Sep 11 18:28:06.081903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.081878    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42db42c7-1079-4b7c-85cb-3b38008da53c" containerName="payload-processing"
Sep 11 18:28:06.081903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.081889    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="42db42c7-1079-4b7c-85cb-3b38008da53c" containerName="payload-processing"
Sep 11 18:28:06.081903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.081902    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="837797d7-5c6e-4a09-8d0f-36bcfd1a0662" containerName="payload-pre-processing"
Sep 11 18:28:06.082011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.081908    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="837797d7-5c6e-4a09-8d0f-36bcfd1a0662" containerName="payload-pre-processing"
Sep 11 18:28:06.082011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.081917    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c43ffc6d-9eae-4ab7-aa27-90e0abfc11af" containerName="revoke-keys"
Sep 11 18:28:06.082011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.081923    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c43ffc6d-9eae-4ab7-aa27-90e0abfc11af" containerName="revoke-keys"
Sep 11 18:28:06.082011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.081999    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="42db42c7-1079-4b7c-85cb-3b38008da53c" containerName="payload-processing"
Sep 11 18:28:06.082011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.082009    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c43ffc6d-9eae-4ab7-aa27-90e0abfc11af" containerName="revoke-keys"
Sep 11 18:28:06.082159 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.082017    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="837797d7-5c6e-4a09-8d0f-36bcfd1a0662" containerName="payload-pre-processing"
Sep 11 18:28:06.085149 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.085133    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"
Sep 11 18:28:06.087965 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.087946    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:28:06.093812 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.093793    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"]
Sep 11 18:28:06.196728 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.196695    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9fd47f-bc49-403a-abda-a5ab84350726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj\" (UID: \"3f9fd47f-bc49-403a-abda-a5ab84350726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"
Sep 11 18:28:06.297912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.297881    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9fd47f-bc49-403a-abda-a5ab84350726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj\" (UID: \"3f9fd47f-bc49-403a-abda-a5ab84350726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"
Sep 11 18:28:06.298591 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.298574    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9fd47f-bc49-403a-abda-a5ab84350726-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj\" (UID: \"3f9fd47f-bc49-403a-abda-a5ab84350726\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"
Sep 11 18:28:06.397412 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.397347    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"
Sep 11 18:28:06.677248 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:06.677222    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"]
Sep 11 18:28:06.677827 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:28:06.677802    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9fd47f_bc49_403a_abda_a5ab84350726.slice/crio-57434fc3ad88470f6a79204ba55eb4f76898026b7efc0c8aabb638c5f4d5a698 WatchSource:0}: Error finding container 57434fc3ad88470f6a79204ba55eb4f76898026b7efc0c8aabb638c5f4d5a698: Status 404 returned error can't find the container with id 57434fc3ad88470f6a79204ba55eb4f76898026b7efc0c8aabb638c5f4d5a698
Sep 11 18:28:07.294766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:07.294673    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj" event={"ID":"3f9fd47f-bc49-403a-abda-a5ab84350726","Type":"ContainerStarted","Data":"b23e7f31e2f3e0d3acf9220a7b183b5d43fc7ce23a7f0a985e9ea52f62969e01"}
Sep 11 18:28:07.294766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:07.294739    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj" event={"ID":"3f9fd47f-bc49-403a-abda-a5ab84350726","Type":"ContainerStarted","Data":"57434fc3ad88470f6a79204ba55eb4f76898026b7efc0c8aabb638c5f4d5a698"}
Sep 11 18:28:07.312644 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:07.312590    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj" podStartSLOduration=1.312574649 podStartE2EDuration="1.312574649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:07.310709365 +0000 UTC m=+2051.080775729" watchObservedRunningTime="2026-09-11 18:28:07.312574649 +0000 UTC m=+2051.082641030"
Sep 11 18:28:08.302048 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:08.302016    2571 generic.go:356] "Generic (PLEG): container finished" podID="3f9fd47f-bc49-403a-abda-a5ab84350726" containerID="b23e7f31e2f3e0d3acf9220a7b183b5d43fc7ce23a7f0a985e9ea52f62969e01" exitCode=0
Sep 11 18:28:08.302466 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:08.302081    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj" event={"ID":"3f9fd47f-bc49-403a-abda-a5ab84350726","Type":"ContainerDied","Data":"b23e7f31e2f3e0d3acf9220a7b183b5d43fc7ce23a7f0a985e9ea52f62969e01"}
Sep 11 18:28:09.442929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:09.442898    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"
Sep 11 18:28:09.528926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:09.528897    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9fd47f-bc49-403a-abda-a5ab84350726-maas-api-service-ca\") pod \"3f9fd47f-bc49-403a-abda-a5ab84350726\" (UID: \"3f9fd47f-bc49-403a-abda-a5ab84350726\") "
Sep 11 18:28:09.529264 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:09.529243    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9fd47f-bc49-403a-abda-a5ab84350726-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f9fd47f-bc49-403a-abda-a5ab84350726" (UID: "3f9fd47f-bc49-403a-abda-a5ab84350726"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:09.629959 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:09.629880    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f9fd47f-bc49-403a-abda-a5ab84350726-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:28:10.314589 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:10.314556    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"
Sep 11 18:28:10.314757 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:10.314586    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj" event={"ID":"3f9fd47f-bc49-403a-abda-a5ab84350726","Type":"ContainerDied","Data":"57434fc3ad88470f6a79204ba55eb4f76898026b7efc0c8aabb638c5f4d5a698"}
Sep 11 18:28:10.314757 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:10.314636    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57434fc3ad88470f6a79204ba55eb4f76898026b7efc0c8aabb638c5f4d5a698"
Sep 11 18:28:21.750918 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:21.750724    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"]
Sep 11 18:28:21.754542 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:21.751499    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5" podUID="b24d3d9e-b330-4e21-b19b-7a3428e0b8d0" containerName="payload-pre-processing" containerID="cri-o://75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f" gracePeriod=30
Sep 11 18:28:21.912074 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:21.912039    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"]
Sep 11 18:28:21.917035 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:21.917006    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be394b-bhtvj"]
Sep 11 18:28:22.011973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.011890    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:28:22.042598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.042572    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-plugins-config-volume\") pod \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\" (UID: \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\") "
Sep 11 18:28:22.042751 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.042659    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9pcj\" (UniqueName: \"kubernetes.io/projected/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-kube-api-access-s9pcj\") pod \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\" (UID: \"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0\") "
Sep 11 18:28:22.042918 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.042897    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b24d3d9e-b330-4e21-b19b-7a3428e0b8d0" (UID: "b24d3d9e-b330-4e21-b19b-7a3428e0b8d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:22.045009 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.044976    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-kube-api-access-s9pcj" (OuterVolumeSpecName: "kube-api-access-s9pcj") pod "b24d3d9e-b330-4e21-b19b-7a3428e0b8d0" (UID: "b24d3d9e-b330-4e21-b19b-7a3428e0b8d0"). InnerVolumeSpecName "kube-api-access-s9pcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:22.143245 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.143211    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9pcj\" (UniqueName: \"kubernetes.io/projected/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-kube-api-access-s9pcj\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:28:22.143245 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.143238    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:28:22.388257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.388168    2571 generic.go:356] "Generic (PLEG): container finished" podID="b24d3d9e-b330-4e21-b19b-7a3428e0b8d0" containerID="75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f" exitCode=0
Sep 11 18:28:22.388257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.388213    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"
Sep 11 18:28:22.388506 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.388252    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5" event={"ID":"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0","Type":"ContainerDied","Data":"75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f"}
Sep 11 18:28:22.388506 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.388335    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5" event={"ID":"b24d3d9e-b330-4e21-b19b-7a3428e0b8d0","Type":"ContainerDied","Data":"fef0208ea8430084ede6981ab720f947834efbf6227e13b5750d53cd3730ac40"}
Sep 11 18:28:22.388506 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.388364    2571 scope.go:117] "RemoveContainer" containerID="75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f"
Sep 11 18:28:22.399598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.399580    2571 scope.go:117] "RemoveContainer" containerID="75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f"
Sep 11 18:28:22.400067 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:28:22.400040    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f\": container with ID starting with 75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f not found: ID does not exist" containerID="75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f"
Sep 11 18:28:22.400162 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.400076    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f"} err="failed to get container status \"75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f\": rpc error: code = NotFound desc = could not find container \"75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f\": container with ID starting with 75d03c94eee9c08860492313f4fd0893206a4c732833616af10537941fb0568f not found: ID does not exist"
Sep 11 18:28:22.417066 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.417036    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"]
Sep 11 18:28:22.420628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.420607    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be394b-678b7479bb-dx7z5"]
Sep 11 18:28:22.730764 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.730733    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f9fd47f-bc49-403a-abda-a5ab84350726" path="/var/lib/kubelet/pods/3f9fd47f-bc49-403a-abda-a5ab84350726/volumes"
Sep 11 18:28:22.731107 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:22.731095    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24d3d9e-b330-4e21-b19b-7a3428e0b8d0" path="/var/lib/kubelet/pods/b24d3d9e-b330-4e21-b19b-7a3428e0b8d0/volumes"
Sep 11 18:28:57.510846 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:28:57.510816    2571 scope.go:117] "RemoveContainer" containerID="113aafade582efcb3a04f180352cac619dbbc0b94aef0fd82d6ba03d50b70936"
Sep 11 18:29:17.104787 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.104753    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"]
Sep 11 18:29:17.105198 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.105182    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24d3d9e-b330-4e21-b19b-7a3428e0b8d0" containerName="payload-pre-processing"
Sep 11 18:29:17.105242 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.105200    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24d3d9e-b330-4e21-b19b-7a3428e0b8d0" containerName="payload-pre-processing"
Sep 11 18:29:17.105242 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.105210    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f9fd47f-bc49-403a-abda-a5ab84350726" containerName="revoke-keys"
Sep 11 18:29:17.105242 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.105215    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f9fd47f-bc49-403a-abda-a5ab84350726" containerName="revoke-keys"
Sep 11 18:29:17.105371 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.105279    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f9fd47f-bc49-403a-abda-a5ab84350726" containerName="revoke-keys"
Sep 11 18:29:17.105371 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.105287    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24d3d9e-b330-4e21-b19b-7a3428e0b8d0" containerName="payload-pre-processing"
Sep 11 18:29:17.108495 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.108478    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"
Sep 11 18:29:17.111389 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.111366    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:29:17.117445 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.117419    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"]
Sep 11 18:29:17.135150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.135120    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9346a8a3-38f7-41a3-8190-666389746ac9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm\" (UID: \"9346a8a3-38f7-41a3-8190-666389746ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"
Sep 11 18:29:17.236059 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.236022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9346a8a3-38f7-41a3-8190-666389746ac9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm\" (UID: \"9346a8a3-38f7-41a3-8190-666389746ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"
Sep 11 18:29:17.236681 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.236661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9346a8a3-38f7-41a3-8190-666389746ac9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm\" (UID: \"9346a8a3-38f7-41a3-8190-666389746ac9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"
Sep 11 18:29:17.421081 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.421051    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"
Sep 11 18:29:17.699016 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.698988    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"]
Sep 11 18:29:17.699676 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:29:17.699651    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9346a8a3_38f7_41a3_8190_666389746ac9.slice/crio-6d4a8634a0a8f2ea0aae1972aa2db497902956f952b18d8ca72bdd85f66f73ed WatchSource:0}: Error finding container 6d4a8634a0a8f2ea0aae1972aa2db497902956f952b18d8ca72bdd85f66f73ed: Status 404 returned error can't find the container with id 6d4a8634a0a8f2ea0aae1972aa2db497902956f952b18d8ca72bdd85f66f73ed
Sep 11 18:29:17.710541 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:17.710512    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm" event={"ID":"9346a8a3-38f7-41a3-8190-666389746ac9","Type":"ContainerStarted","Data":"6d4a8634a0a8f2ea0aae1972aa2db497902956f952b18d8ca72bdd85f66f73ed"}
Sep 11 18:29:18.719058 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:18.719020    2571 generic.go:356] "Generic (PLEG): container finished" podID="9346a8a3-38f7-41a3-8190-666389746ac9" containerID="7fb7ef01b861544cb6c01e1d4dbe09ac5334ded174418ff6e3839878588762bd" exitCode=0
Sep 11 18:29:18.719486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:18.719080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm" event={"ID":"9346a8a3-38f7-41a3-8190-666389746ac9","Type":"ContainerDied","Data":"7fb7ef01b861544cb6c01e1d4dbe09ac5334ded174418ff6e3839878588762bd"}
Sep 11 18:29:19.864498 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:19.864473    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"
Sep 11 18:29:19.961078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:19.961047    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9346a8a3-38f7-41a3-8190-666389746ac9-maas-api-service-ca\") pod \"9346a8a3-38f7-41a3-8190-666389746ac9\" (UID: \"9346a8a3-38f7-41a3-8190-666389746ac9\") "
Sep 11 18:29:19.961461 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:19.961440    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9346a8a3-38f7-41a3-8190-666389746ac9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9346a8a3-38f7-41a3-8190-666389746ac9" (UID: "9346a8a3-38f7-41a3-8190-666389746ac9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:20.062700 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:20.062616    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9346a8a3-38f7-41a3-8190-666389746ac9-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:20.732692 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:20.732661    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"
Sep 11 18:29:20.732876 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:20.732736    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm" event={"ID":"9346a8a3-38f7-41a3-8190-666389746ac9","Type":"ContainerDied","Data":"6d4a8634a0a8f2ea0aae1972aa2db497902956f952b18d8ca72bdd85f66f73ed"}
Sep 11 18:29:20.732876 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:20.732776    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d4a8634a0a8f2ea0aae1972aa2db497902956f952b18d8ca72bdd85f66f73ed"
Sep 11 18:29:27.779936 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:27.779899    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"]
Sep 11 18:29:27.780414 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:27.780159    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz" podUID="27b513bd-d35f-478f-aa45-69a05b126219" containerName="payload-processing" containerID="cri-o://584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497" gracePeriod=30
Sep 11 18:29:27.784493 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:27.784463    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"]
Sep 11 18:29:27.784812 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:27.784712    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl" podUID="a2c08d5f-9e82-476d-8c9a-763f35087537" containerName="payload-pre-processing" containerID="cri-o://ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151" gracePeriod=30
Sep 11 18:29:27.962554 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:27.962474    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"]
Sep 11 18:29:27.966598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:27.966554    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-74a302-m26sm"]
Sep 11 18:29:28.065732 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.065705    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:29:28.069405 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.069361    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:29:28.131321 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.131273    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b513bd-d35f-478f-aa45-69a05b126219-plugins-config-volume\") pod \"27b513bd-d35f-478f-aa45-69a05b126219\" (UID: \"27b513bd-d35f-478f-aa45-69a05b126219\") "
Sep 11 18:29:28.131512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.131366    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c08d5f-9e82-476d-8c9a-763f35087537-plugins-config-volume\") pod \"a2c08d5f-9e82-476d-8c9a-763f35087537\" (UID: \"a2c08d5f-9e82-476d-8c9a-763f35087537\") "
Sep 11 18:29:28.131512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.131404    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sfvd\" (UniqueName: \"kubernetes.io/projected/27b513bd-d35f-478f-aa45-69a05b126219-kube-api-access-7sfvd\") pod \"27b513bd-d35f-478f-aa45-69a05b126219\" (UID: \"27b513bd-d35f-478f-aa45-69a05b126219\") "
Sep 11 18:29:28.131512 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.131444    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j4xn\" (UniqueName: \"kubernetes.io/projected/a2c08d5f-9e82-476d-8c9a-763f35087537-kube-api-access-2j4xn\") pod \"a2c08d5f-9e82-476d-8c9a-763f35087537\" (UID: \"a2c08d5f-9e82-476d-8c9a-763f35087537\") "
Sep 11 18:29:28.131734 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.131706    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b513bd-d35f-478f-aa45-69a05b126219-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27b513bd-d35f-478f-aa45-69a05b126219" (UID: "27b513bd-d35f-478f-aa45-69a05b126219"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:28.132070 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.132049    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c08d5f-9e82-476d-8c9a-763f35087537-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2c08d5f-9e82-476d-8c9a-763f35087537" (UID: "a2c08d5f-9e82-476d-8c9a-763f35087537"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:28.133639 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.133614    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c08d5f-9e82-476d-8c9a-763f35087537-kube-api-access-2j4xn" (OuterVolumeSpecName: "kube-api-access-2j4xn") pod "a2c08d5f-9e82-476d-8c9a-763f35087537" (UID: "a2c08d5f-9e82-476d-8c9a-763f35087537"). InnerVolumeSpecName "kube-api-access-2j4xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:28.133920 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.133903    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b513bd-d35f-478f-aa45-69a05b126219-kube-api-access-7sfvd" (OuterVolumeSpecName: "kube-api-access-7sfvd") pod "27b513bd-d35f-478f-aa45-69a05b126219" (UID: "27b513bd-d35f-478f-aa45-69a05b126219"). InnerVolumeSpecName "kube-api-access-7sfvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:28.232641 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.232602    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j4xn\" (UniqueName: \"kubernetes.io/projected/a2c08d5f-9e82-476d-8c9a-763f35087537-kube-api-access-2j4xn\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:28.232641 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.232634    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27b513bd-d35f-478f-aa45-69a05b126219-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:28.232641 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.232643    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c08d5f-9e82-476d-8c9a-763f35087537-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:28.232888 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.232652    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sfvd\" (UniqueName: \"kubernetes.io/projected/27b513bd-d35f-478f-aa45-69a05b126219-kube-api-access-7sfvd\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:28.731185 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.731147    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9346a8a3-38f7-41a3-8190-666389746ac9" path="/var/lib/kubelet/pods/9346a8a3-38f7-41a3-8190-666389746ac9/volumes"
Sep 11 18:29:28.789165 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.789121    2571 generic.go:356] "Generic (PLEG): container finished" podID="27b513bd-d35f-478f-aa45-69a05b126219" containerID="584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497" exitCode=0
Sep 11 18:29:28.789671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.789182    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"
Sep 11 18:29:28.789671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.789217    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz" event={"ID":"27b513bd-d35f-478f-aa45-69a05b126219","Type":"ContainerDied","Data":"584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497"}
Sep 11 18:29:28.789671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.789332    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz" event={"ID":"27b513bd-d35f-478f-aa45-69a05b126219","Type":"ContainerDied","Data":"35174de41ba06088f8680a6ae7bba4a6e423805de0200bf977b72f90408c685e"}
Sep 11 18:29:28.789671 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.789353    2571 scope.go:117] "RemoveContainer" containerID="584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497"
Sep 11 18:29:28.790936 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.790848    2571 generic.go:356] "Generic (PLEG): container finished" podID="a2c08d5f-9e82-476d-8c9a-763f35087537" containerID="ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151" exitCode=0
Sep 11 18:29:28.790936 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.790929    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl" event={"ID":"a2c08d5f-9e82-476d-8c9a-763f35087537","Type":"ContainerDied","Data":"ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151"}
Sep 11 18:29:28.791067 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.790988    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"
Sep 11 18:29:28.791205 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.790989    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl" event={"ID":"a2c08d5f-9e82-476d-8c9a-763f35087537","Type":"ContainerDied","Data":"b8356d2397c0490571a896ea7af7b08e4b1b495d2005cee77d43ed417b67f8df"}
Sep 11 18:29:28.805020 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.804998    2571 scope.go:117] "RemoveContainer" containerID="584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497"
Sep 11 18:29:28.805361 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:29:28.805331    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497\": container with ID starting with 584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497 not found: ID does not exist" containerID="584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497"
Sep 11 18:29:28.805440 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.805356    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497"} err="failed to get container status \"584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497\": rpc error: code = NotFound desc = could not find container \"584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497\": container with ID starting with 584b5c5cd432995c908f1cc2fa7a587a08d45f1e8fafe08654e5e39b9be48497 not found: ID does not exist"
Sep 11 18:29:28.805440 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.805372    2571 scope.go:117] "RemoveContainer" containerID="ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151"
Sep 11 18:29:28.813578 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.813560    2571 scope.go:117] "RemoveContainer" containerID="ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151"
Sep 11 18:29:28.813836 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:29:28.813818    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151\": container with ID starting with ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151 not found: ID does not exist" containerID="ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151"
Sep 11 18:29:28.813898 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.813841    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151"} err="failed to get container status \"ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151\": rpc error: code = NotFound desc = could not find container \"ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151\": container with ID starting with ebb607e566c1a969b8220b0d837b7bde982575be068a60e4746fe6a256db9151 not found: ID does not exist"
Sep 11 18:29:28.817375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.817354    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"]
Sep 11 18:29:28.820926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.820897    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-74a302-796f97bd6b-g8vpl"]
Sep 11 18:29:28.831515 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.831495    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"]
Sep 11 18:29:28.834870 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:28.834848    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-74a302-86bdb8487c-kz5zz"]
Sep 11 18:29:30.731548 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:30.731515    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b513bd-d35f-478f-aa45-69a05b126219" path="/var/lib/kubelet/pods/27b513bd-d35f-478f-aa45-69a05b126219/volumes"
Sep 11 18:29:30.731957 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:30.731887    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c08d5f-9e82-476d-8c9a-763f35087537" path="/var/lib/kubelet/pods/a2c08d5f-9e82-476d-8c9a-763f35087537/volumes"
Sep 11 18:29:34.780010 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:34.779974    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"]
Sep 11 18:29:34.780488 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:34.780222    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy" containerID="cri-o://e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a" gracePeriod=30
Sep 11 18:29:35.142532 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142457    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"]
Sep 11 18:29:35.142858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142846    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c08d5f-9e82-476d-8c9a-763f35087537" containerName="payload-pre-processing"
Sep 11 18:29:35.142903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142860    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c08d5f-9e82-476d-8c9a-763f35087537" containerName="payload-pre-processing"
Sep 11 18:29:35.142903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142876    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b513bd-d35f-478f-aa45-69a05b126219" containerName="payload-processing"
Sep 11 18:29:35.142903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142881    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b513bd-d35f-478f-aa45-69a05b126219" containerName="payload-processing"
Sep 11 18:29:35.142903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142893    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9346a8a3-38f7-41a3-8190-666389746ac9" containerName="revoke-keys"
Sep 11 18:29:35.142903 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142898    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9346a8a3-38f7-41a3-8190-666389746ac9" containerName="revoke-keys"
Sep 11 18:29:35.143056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142962    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b513bd-d35f-478f-aa45-69a05b126219" containerName="payload-processing"
Sep 11 18:29:35.143056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142969    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9346a8a3-38f7-41a3-8190-666389746ac9" containerName="revoke-keys"
Sep 11 18:29:35.143056 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.142974    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c08d5f-9e82-476d-8c9a-763f35087537" containerName="payload-pre-processing"
Sep 11 18:29:35.145825 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.145810    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"
Sep 11 18:29:35.148606 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.148575    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:29:35.154968 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.154525    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"]
Sep 11 18:29:35.299602 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.299564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/700dce69-e898-42b3-a443-ff7ffdb0bb5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm\" (UID: \"700dce69-e898-42b3-a443-ff7ffdb0bb5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"
Sep 11 18:29:35.400560 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.400478    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/700dce69-e898-42b3-a443-ff7ffdb0bb5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm\" (UID: \"700dce69-e898-42b3-a443-ff7ffdb0bb5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"
Sep 11 18:29:35.401130 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.401106    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/700dce69-e898-42b3-a443-ff7ffdb0bb5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm\" (UID: \"700dce69-e898-42b3-a443-ff7ffdb0bb5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"
Sep 11 18:29:35.459018 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.458985    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"
Sep 11 18:29:35.735111 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.735081    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"]
Sep 11 18:29:35.735347 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:29:35.735316    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700dce69_e898_42b3_a443_ff7ffdb0bb5f.slice/crio-7d46b65b6b5f4f5b2776e152c533e4dd254a065073998ad9fadbb6846eab6039 WatchSource:0}: Error finding container 7d46b65b6b5f4f5b2776e152c533e4dd254a065073998ad9fadbb6846eab6039: Status 404 returned error can't find the container with id 7d46b65b6b5f4f5b2776e152c533e4dd254a065073998ad9fadbb6846eab6039
Sep 11 18:29:35.840547 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:35.840510    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm" event={"ID":"700dce69-e898-42b3-a443-ff7ffdb0bb5f","Type":"ContainerStarted","Data":"7d46b65b6b5f4f5b2776e152c533e4dd254a065073998ad9fadbb6846eab6039"}
Sep 11 18:29:36.847850 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:36.847763    2571 generic.go:356] "Generic (PLEG): container finished" podID="700dce69-e898-42b3-a443-ff7ffdb0bb5f" containerID="2c6df278560f7978eeb6e09197c022a35e5d2c6e334903710a99e5f23741681a" exitCode=0
Sep 11 18:29:36.848224 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:36.847848    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm" event={"ID":"700dce69-e898-42b3-a443-ff7ffdb0bb5f","Type":"ContainerDied","Data":"2c6df278560f7978eeb6e09197c022a35e5d2c6e334903710a99e5f23741681a"}
Sep 11 18:29:37.984492 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:37.984467    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"
Sep 11 18:29:38.125463 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:38.125383    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/700dce69-e898-42b3-a443-ff7ffdb0bb5f-maas-api-service-ca\") pod \"700dce69-e898-42b3-a443-ff7ffdb0bb5f\" (UID: \"700dce69-e898-42b3-a443-ff7ffdb0bb5f\") "
Sep 11 18:29:38.125728 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:38.125708    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/700dce69-e898-42b3-a443-ff7ffdb0bb5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "700dce69-e898-42b3-a443-ff7ffdb0bb5f" (UID: "700dce69-e898-42b3-a443-ff7ffdb0bb5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:38.226269 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:38.226237    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/700dce69-e898-42b3-a443-ff7ffdb0bb5f-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:38.862354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:38.862313    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm" event={"ID":"700dce69-e898-42b3-a443-ff7ffdb0bb5f","Type":"ContainerDied","Data":"7d46b65b6b5f4f5b2776e152c533e4dd254a065073998ad9fadbb6846eab6039"}
Sep 11 18:29:38.862354 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:38.862342    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"
Sep 11 18:29:38.862564 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:38.862368    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d46b65b6b5f4f5b2776e152c533e4dd254a065073998ad9fadbb6846eab6039"
Sep 11 18:29:40.034429 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.034405    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:29:40.143820 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143737    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-envoy\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.143820 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143775    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-podinfo\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.143820 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143796    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istiod-ca-cert\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.143820 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143812    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-token\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.144150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143831    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-credential-socket\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.144150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143897    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-socket\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.144150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143928    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-data\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.144150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143954    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-certs\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.144150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.143976    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvcc6\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-kube-api-access-gvcc6\") pod \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\" (UID: \"2693f0ee-2647-4c60-b4f2-5937f1e856a0\") "
Sep 11 18:29:40.144458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.144193    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:40.144458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.144251    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:40.144458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.144321    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.144458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.144267    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:40.144458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.144389    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:40.144458 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.144440    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:40.152417 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.152281    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:29:40.152704 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.152607    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:40.152704 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.152625    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:40.152996 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.152975    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-kube-api-access-gvcc6" (OuterVolumeSpecName: "kube-api-access-gvcc6") pod "2693f0ee-2647-4c60-b4f2-5937f1e856a0" (UID: "2693f0ee-2647-4c60-b4f2-5937f1e856a0"). InnerVolumeSpecName "kube-api-access-gvcc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:40.246107 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.246071    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.246107 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.246106    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.246352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.246121    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.246352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.246163    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvcc6\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-kube-api-access-gvcc6\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.246352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.246177    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.246352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.246191    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.246352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.246203    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.246352 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.246216    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2693f0ee-2647-4c60-b4f2-5937f1e856a0-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:40.875777 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.875746    2571 generic.go:356] "Generic (PLEG): container finished" podID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerID="e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a" exitCode=0
Sep 11 18:29:40.875956 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.875810    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"
Sep 11 18:29:40.875956 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.875833    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" event={"ID":"2693f0ee-2647-4c60-b4f2-5937f1e856a0","Type":"ContainerDied","Data":"e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a"}
Sep 11 18:29:40.875956 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.875888    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz" event={"ID":"2693f0ee-2647-4c60-b4f2-5937f1e856a0","Type":"ContainerDied","Data":"83d3bc9c23583ca3d96973eec0b29ded13f124a261eb2958a686f2f224f8eb04"}
Sep 11 18:29:40.875956 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.875911    2571 scope.go:117] "RemoveContainer" containerID="e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a"
Sep 11 18:29:40.885315 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.885280    2571 scope.go:117] "RemoveContainer" containerID="74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5"
Sep 11 18:29:40.894222 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.894198    2571 scope.go:117] "RemoveContainer" containerID="e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a"
Sep 11 18:29:40.894499 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:29:40.894482    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a\": container with ID starting with e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a not found: ID does not exist" containerID="e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a"
Sep 11 18:29:40.894562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.894506    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a"} err="failed to get container status \"e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a\": rpc error: code = NotFound desc = could not find container \"e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a\": container with ID starting with e092412ed383f0157d139d8583ce3b5199a3adb4b51cc64d5ad36a7017589a6a not found: ID does not exist"
Sep 11 18:29:40.894562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.894521    2571 scope.go:117] "RemoveContainer" containerID="74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5"
Sep 11 18:29:40.894785 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:29:40.894771    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5\": container with ID starting with 74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5 not found: ID does not exist" containerID="74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5"
Sep 11 18:29:40.894828 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.894788    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5"} err="failed to get container status \"74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5\": rpc error: code = NotFound desc = could not find container \"74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5\": container with ID starting with 74dc80454429a67afd6caf15a27507c371374fe03bec7394a5cb351ef5fe74a5 not found: ID does not exist"
Sep 11 18:29:40.895381 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.895362    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"]
Sep 11 18:29:40.898687 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:40.898669    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-74a302-openshift-default-76997f94d6-gj8rz"]
Sep 11 18:29:42.730245 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:42.730210    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" path="/var/lib/kubelet/pods/2693f0ee-2647-4c60-b4f2-5937f1e856a0/volumes"
Sep 11 18:29:46.084390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.084358    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"]
Sep 11 18:29:46.084860 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.084594    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956" podUID="73702daf-d255-4e06-8dd0-7956fd128508" containerName="payload-processing" containerID="cri-o://714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f" gracePeriod=30
Sep 11 18:29:46.091556 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.091528    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"]
Sep 11 18:29:46.091940 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.091907    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r" podUID="ac7a6760-f70c-431d-86fe-98f082668837" containerName="payload-pre-processing" containerID="cri-o://5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5" gracePeriod=30
Sep 11 18:29:46.244383 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.244355    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"]
Sep 11 18:29:46.246949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.246924    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-a5bbf6-l2wnm"]
Sep 11 18:29:46.370796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.370772    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:29:46.372744 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.372726    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:29:46.500986 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.500955    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxb6w\" (UniqueName: \"kubernetes.io/projected/73702daf-d255-4e06-8dd0-7956fd128508-kube-api-access-xxb6w\") pod \"73702daf-d255-4e06-8dd0-7956fd128508\" (UID: \"73702daf-d255-4e06-8dd0-7956fd128508\") "
Sep 11 18:29:46.501169 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.501008    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpmvg\" (UniqueName: \"kubernetes.io/projected/ac7a6760-f70c-431d-86fe-98f082668837-kube-api-access-qpmvg\") pod \"ac7a6760-f70c-431d-86fe-98f082668837\" (UID: \"ac7a6760-f70c-431d-86fe-98f082668837\") "
Sep 11 18:29:46.501169 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.501035    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7a6760-f70c-431d-86fe-98f082668837-plugins-config-volume\") pod \"ac7a6760-f70c-431d-86fe-98f082668837\" (UID: \"ac7a6760-f70c-431d-86fe-98f082668837\") "
Sep 11 18:29:46.501169 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.501076    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73702daf-d255-4e06-8dd0-7956fd128508-plugins-config-volume\") pod \"73702daf-d255-4e06-8dd0-7956fd128508\" (UID: \"73702daf-d255-4e06-8dd0-7956fd128508\") "
Sep 11 18:29:46.501460 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.501434    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7a6760-f70c-431d-86fe-98f082668837-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac7a6760-f70c-431d-86fe-98f082668837" (UID: "ac7a6760-f70c-431d-86fe-98f082668837"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:46.501549 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.501523    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73702daf-d255-4e06-8dd0-7956fd128508-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73702daf-d255-4e06-8dd0-7956fd128508" (UID: "73702daf-d255-4e06-8dd0-7956fd128508"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:46.503173 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.503155    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7a6760-f70c-431d-86fe-98f082668837-kube-api-access-qpmvg" (OuterVolumeSpecName: "kube-api-access-qpmvg") pod "ac7a6760-f70c-431d-86fe-98f082668837" (UID: "ac7a6760-f70c-431d-86fe-98f082668837"). InnerVolumeSpecName "kube-api-access-qpmvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:46.503240 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.503181    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73702daf-d255-4e06-8dd0-7956fd128508-kube-api-access-xxb6w" (OuterVolumeSpecName: "kube-api-access-xxb6w") pod "73702daf-d255-4e06-8dd0-7956fd128508" (UID: "73702daf-d255-4e06-8dd0-7956fd128508"). InnerVolumeSpecName "kube-api-access-xxb6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:46.601964 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.601895    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73702daf-d255-4e06-8dd0-7956fd128508-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:46.601964 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.601921    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxb6w\" (UniqueName: \"kubernetes.io/projected/73702daf-d255-4e06-8dd0-7956fd128508-kube-api-access-xxb6w\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:46.601964 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.601930    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpmvg\" (UniqueName: \"kubernetes.io/projected/ac7a6760-f70c-431d-86fe-98f082668837-kube-api-access-qpmvg\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:46.601964 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.601939    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7a6760-f70c-431d-86fe-98f082668837-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:46.730690 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.730659    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="700dce69-e898-42b3-a443-ff7ffdb0bb5f" path="/var/lib/kubelet/pods/700dce69-e898-42b3-a443-ff7ffdb0bb5f/volumes"
Sep 11 18:29:46.913527 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.913494    2571 generic.go:356] "Generic (PLEG): container finished" podID="73702daf-d255-4e06-8dd0-7956fd128508" containerID="714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f" exitCode=0
Sep 11 18:29:46.913703 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.913539    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"
Sep 11 18:29:46.913703 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.913589    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956" event={"ID":"73702daf-d255-4e06-8dd0-7956fd128508","Type":"ContainerDied","Data":"714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f"}
Sep 11 18:29:46.913703 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.913642    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956" event={"ID":"73702daf-d255-4e06-8dd0-7956fd128508","Type":"ContainerDied","Data":"1983cc12354c44fff87fd7ee535863938f8250d2d84baa9317168bc1dec648a3"}
Sep 11 18:29:46.913703 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.913671    2571 scope.go:117] "RemoveContainer" containerID="714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f"
Sep 11 18:29:46.915212 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.915189    2571 generic.go:356] "Generic (PLEG): container finished" podID="ac7a6760-f70c-431d-86fe-98f082668837" containerID="5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5" exitCode=0
Sep 11 18:29:46.915337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.915235    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"
Sep 11 18:29:46.915337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.915244    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r" event={"ID":"ac7a6760-f70c-431d-86fe-98f082668837","Type":"ContainerDied","Data":"5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5"}
Sep 11 18:29:46.915337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.915287    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r" event={"ID":"ac7a6760-f70c-431d-86fe-98f082668837","Type":"ContainerDied","Data":"0c8a5fc1591675fdcd673f20d183206f8dc561cbe1d6b1e320bb31bd07ba7bc4"}
Sep 11 18:29:46.923648 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.923631    2571 scope.go:117] "RemoveContainer" containerID="714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f"
Sep 11 18:29:46.923898 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:29:46.923878    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f\": container with ID starting with 714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f not found: ID does not exist" containerID="714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f"
Sep 11 18:29:46.923965 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.923905    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f"} err="failed to get container status \"714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f\": rpc error: code = NotFound desc = could not find container \"714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f\": container with ID starting with 714b9ea684d6f4e69d4ed9b5e5363de46325a53cc340958edff8c664a92bb79f not found: ID does not exist"
Sep 11 18:29:46.923965 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.923921    2571 scope.go:117] "RemoveContainer" containerID="5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5"
Sep 11 18:29:46.931760 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.931743    2571 scope.go:117] "RemoveContainer" containerID="5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5"
Sep 11 18:29:46.932009 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:29:46.931992    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5\": container with ID starting with 5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5 not found: ID does not exist" containerID="5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5"
Sep 11 18:29:46.932057 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.932014    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5"} err="failed to get container status \"5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5\": rpc error: code = NotFound desc = could not find container \"5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5\": container with ID starting with 5466d61d97a6dcdc00cc168c32708ef6fcdba41f773bb4be6ec2f2dde6c2a9e5 not found: ID does not exist"
Sep 11 18:29:46.936318 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.936281    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"]
Sep 11 18:29:46.939689 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.939670    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a5bbf6-c47bcb859-pg956"]
Sep 11 18:29:46.952135 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.952114    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"]
Sep 11 18:29:46.956250 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:46.956227    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a5bbf6-65f76fb646-v8r4r"]
Sep 11 18:29:48.730808 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:48.730778    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73702daf-d255-4e06-8dd0-7956fd128508" path="/var/lib/kubelet/pods/73702daf-d255-4e06-8dd0-7956fd128508/volumes"
Sep 11 18:29:48.731161 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:48.731128    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac7a6760-f70c-431d-86fe-98f082668837" path="/var/lib/kubelet/pods/ac7a6760-f70c-431d-86fe-98f082668837/volumes"
Sep 11 18:29:53.456076 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.456042    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"]
Sep 11 18:29:53.456569 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.456237    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy" containerID="cri-o://f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3" gracePeriod=30
Sep 11 18:29:53.745228 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745149    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"]
Sep 11 18:29:53.745604 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745588    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy"
Sep 11 18:29:53.745668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745607    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy"
Sep 11 18:29:53.745668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745615    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy"
Sep 11 18:29:53.745668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745621    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy"
Sep 11 18:29:53.745668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745629    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac7a6760-f70c-431d-86fe-98f082668837" containerName="payload-pre-processing"
Sep 11 18:29:53.745668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745635    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac7a6760-f70c-431d-86fe-98f082668837" containerName="payload-pre-processing"
Sep 11 18:29:53.745668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745646    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="700dce69-e898-42b3-a443-ff7ffdb0bb5f" containerName="revoke-keys"
Sep 11 18:29:53.745668 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745651    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="700dce69-e898-42b3-a443-ff7ffdb0bb5f" containerName="revoke-keys"
Sep 11 18:29:53.745881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745675    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73702daf-d255-4e06-8dd0-7956fd128508" containerName="payload-processing"
Sep 11 18:29:53.745881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745686    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="73702daf-d255-4e06-8dd0-7956fd128508" containerName="payload-processing"
Sep 11 18:29:53.745881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745765    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy"
Sep 11 18:29:53.745881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745777    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2693f0ee-2647-4c60-b4f2-5937f1e856a0" containerName="istio-proxy"
Sep 11 18:29:53.745881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745786    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac7a6760-f70c-431d-86fe-98f082668837" containerName="payload-pre-processing"
Sep 11 18:29:53.745881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745794    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="73702daf-d255-4e06-8dd0-7956fd128508" containerName="payload-processing"
Sep 11 18:29:53.745881 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.745801    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="700dce69-e898-42b3-a443-ff7ffdb0bb5f" containerName="revoke-keys"
Sep 11 18:29:53.750429 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.750409    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"
Sep 11 18:29:53.753424 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.753399    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:29:53.755734 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.755651    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"]
Sep 11 18:29:53.759913 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.759629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f226e3e7-d39f-4ae4-9a56-294afd9c1f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n\" (UID: \"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"
Sep 11 18:29:53.860741 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.860709    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f226e3e7-d39f-4ae4-9a56-294afd9c1f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n\" (UID: \"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"
Sep 11 18:29:53.861386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:53.861368    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f226e3e7-d39f-4ae4-9a56-294afd9c1f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n\" (UID: \"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"
Sep 11 18:29:54.064413 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:54.064329    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"
Sep 11 18:29:54.335961 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:54.335933    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"]
Sep 11 18:29:54.336656 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:29:54.336633    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf226e3e7_d39f_4ae4_9a56_294afd9c1f9b.slice/crio-7174da158d8653d91e3fe19913d56f5d71a25795d2615243693827120878ff3b WatchSource:0}: Error finding container 7174da158d8653d91e3fe19913d56f5d71a25795d2615243693827120878ff3b: Status 404 returned error can't find the container with id 7174da158d8653d91e3fe19913d56f5d71a25795d2615243693827120878ff3b
Sep 11 18:29:54.966858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:54.966775    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n" event={"ID":"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b","Type":"ContainerStarted","Data":"801b61cb25cb161691e933dca6b8c8841261041633084d1e41935ca64184c550"}
Sep 11 18:29:54.966858 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:54.966827    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n" event={"ID":"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b","Type":"ContainerStarted","Data":"7174da158d8653d91e3fe19913d56f5d71a25795d2615243693827120878ff3b"}
Sep 11 18:29:54.985829 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:54.985781    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n" podStartSLOduration=1.985767931 podStartE2EDuration="1.985767931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:54.983843881 +0000 UTC m=+2158.753910229" watchObservedRunningTime="2026-09-11 18:29:54.985767931 +0000 UTC m=+2158.755834279"
Sep 11 18:29:55.974773 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:55.974739    2571 generic.go:356] "Generic (PLEG): container finished" podID="f226e3e7-d39f-4ae4-9a56-294afd9c1f9b" containerID="801b61cb25cb161691e933dca6b8c8841261041633084d1e41935ca64184c550" exitCode=0
Sep 11 18:29:55.975215 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:55.974824    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n" event={"ID":"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b","Type":"ContainerDied","Data":"801b61cb25cb161691e933dca6b8c8841261041633084d1e41935ca64184c550"}
Sep 11 18:29:57.115065 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:57.115042    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"
Sep 11 18:29:57.188206 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:57.188168    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f226e3e7-d39f-4ae4-9a56-294afd9c1f9b-maas-api-service-ca\") pod \"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b\" (UID: \"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b\") "
Sep 11 18:29:57.188557 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:57.188536    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f226e3e7-d39f-4ae4-9a56-294afd9c1f9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f226e3e7-d39f-4ae4-9a56-294afd9c1f9b" (UID: "f226e3e7-d39f-4ae4-9a56-294afd9c1f9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:57.289362 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:57.289269    2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f226e3e7-d39f-4ae4-9a56-294afd9c1f9b-maas-api-service-ca\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:57.987698 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:57.987670    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"
Sep 11 18:29:57.987895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:57.987670    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n" event={"ID":"f226e3e7-d39f-4ae4-9a56-294afd9c1f9b","Type":"ContainerDied","Data":"7174da158d8653d91e3fe19913d56f5d71a25795d2615243693827120878ff3b"}
Sep 11 18:29:57.987895 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:57.987809    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7174da158d8653d91e3fe19913d56f5d71a25795d2615243693827120878ff3b"
Sep 11 18:29:58.328337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.328237    2571 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:29:58.328337 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.328320    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:29:58.716126 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.716096    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:29:58.799772 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799735    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-certs\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.799772 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799770    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-socket\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.800032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799798    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-envoy\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.800032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799820    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-data\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.800032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799841    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-token\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.800032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799876    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istiod-ca-cert\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.800032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799895    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzgpt\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-kube-api-access-kzgpt\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.800032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799926    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-podinfo\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.800032 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.799955    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-credential-socket\") pod \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\" (UID: \"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b\") "
Sep 11 18:29:58.800428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800053    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:58.800428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800109    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:58.800428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800229    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:58.800428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800293    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:58.800428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800373    2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.800428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800391    2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-workload-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.800428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800390    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:58.800428 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800405    2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-data\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.800778 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.800439    2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-credential-socket\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.802003 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.801975    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:29:58.802152 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.802134    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:58.802486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.802467    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-kube-api-access-kzgpt" (OuterVolumeSpecName: "kube-api-access-kzgpt") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "kube-api-access-kzgpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:58.802486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.802477    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" (UID: "2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:29:58.901290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.901211    2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-token\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.901290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.901237    2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istiod-ca-cert\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.901290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.901247    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzgpt\" (UniqueName: \"kubernetes.io/projected/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-kube-api-access-kzgpt\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.901290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.901255    2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-podinfo\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.901290 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.901265    2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b-istio-envoy\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:29:58.995200 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.995172    2571 generic.go:356] "Generic (PLEG): container finished" podID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerID="f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3" exitCode=0
Sep 11 18:29:58.995390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.995220    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"
Sep 11 18:29:58.995390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.995240    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" event={"ID":"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b","Type":"ContainerDied","Data":"f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3"}
Sep 11 18:29:58.995390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.995284    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l" event={"ID":"2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b","Type":"ContainerDied","Data":"ca9e8dab9ea9096fd7677aced857548b75c5a7c791d420c4be3af2de1cbf1e9e"}
Sep 11 18:29:58.995390 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:58.995333    2571 scope.go:117] "RemoveContainer" containerID="f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3"
Sep 11 18:29:59.009707 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:59.009687    2571 scope.go:117] "RemoveContainer" containerID="7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f"
Sep 11 18:29:59.018019 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:59.017997    2571 scope.go:117] "RemoveContainer" containerID="f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3"
Sep 11 18:29:59.018309 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:29:59.018273    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3\": container with ID starting with f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3 not found: ID does not exist" containerID="f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3"
Sep 11 18:29:59.018365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:59.018317    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3"} err="failed to get container status \"f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3\": rpc error: code = NotFound desc = could not find container \"f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3\": container with ID starting with f52b72f0e39bdc7f842b5158223bc6fbb21f4916f6dffbd443f941e9ec16c5b3 not found: ID does not exist"
Sep 11 18:29:59.018365 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:59.018340    2571 scope.go:117] "RemoveContainer" containerID="7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f"
Sep 11 18:29:59.018613 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:29:59.018594    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f\": container with ID starting with 7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f not found: ID does not exist" containerID="7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f"
Sep 11 18:29:59.018667 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:59.018617    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f"} err="failed to get container status \"7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f\": rpc error: code = NotFound desc = could not find container \"7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f\": container with ID starting with 7c1358772bc178bf9dabc0b18e4ee6b88d6c440ea10c5c8b4943295f3f926a2f not found: ID does not exist"
Sep 11 18:29:59.025174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:59.025146    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"]
Sep 11 18:29:59.027767 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:29:59.027748    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a5bbf6-openshift-default-55f47b9dd8-8fj6l"]
Sep 11 18:30:00.134631 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.134595    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7"]
Sep 11 18:30:00.135182 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135163    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy"
Sep 11 18:30:00.135231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135185    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy"
Sep 11 18:30:00.135231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135198    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f226e3e7-d39f-4ae4-9a56-294afd9c1f9b" containerName="revoke-keys"
Sep 11 18:30:00.135231 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135207    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f226e3e7-d39f-4ae4-9a56-294afd9c1f9b" containerName="revoke-keys"
Sep 11 18:30:00.135356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135230    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy"
Sep 11 18:30:00.135356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135239    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy"
Sep 11 18:30:00.135356 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135343    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f226e3e7-d39f-4ae4-9a56-294afd9c1f9b" containerName="revoke-keys"
Sep 11 18:30:00.135454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135360    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy"
Sep 11 18:30:00.135454 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.135370    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" containerName="istio-proxy"
Sep 11 18:30:00.140090 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.140070    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"]
Sep 11 18:30:00.140240 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.140214    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7"
Sep 11 18:30:00.143425 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.143404    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"
Sep 11 18:30:00.143954 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.143933    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:30:00.146155 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.146130    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"]
Sep 11 18:30:00.148989 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.148969    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7"]
Sep 11 18:30:00.155394 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.155374    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7"
Sep 11 18:30:00.162802 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.162783    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"
Sep 11 18:30:00.430839 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.430810    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7"]
Sep 11 18:30:00.432276 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:30:00.432255    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711fa7dc_b20c_4cf1_be20_e7ff3ae94c45.slice/crio-5c78bcb020818f15d3ce933c042b81d26bca75137c53a295ffa3556561b8f5dd WatchSource:0}: Error finding container 5c78bcb020818f15d3ce933c042b81d26bca75137c53a295ffa3556561b8f5dd: Status 404 returned error can't find the container with id 5c78bcb020818f15d3ce933c042b81d26bca75137c53a295ffa3556561b8f5dd
Sep 11 18:30:00.440408 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.440385    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"]
Sep 11 18:30:00.441646 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:30:00.441627    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e77659c_ef34_4072_bb36_4fee9101b297.slice/crio-bda7752f8a10faf2ed656f94ed28e838702bb908bc90c5441ac588d1c6f88c1d WatchSource:0}: Error finding container bda7752f8a10faf2ed656f94ed28e838702bb908bc90c5441ac588d1c6f88c1d: Status 404 returned error can't find the container with id bda7752f8a10faf2ed656f94ed28e838702bb908bc90c5441ac588d1c6f88c1d
Sep 11 18:30:00.730659 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:00.730596    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b" path="/var/lib/kubelet/pods/2557dc4d-7ea1-4bad-ac5a-44e21d0a4b0b/volumes"
Sep 11 18:30:01.010455 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:01.010364    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp" event={"ID":"4e77659c-ef34-4072-bb36-4fee9101b297","Type":"ContainerStarted","Data":"8941883c0c272db08f5a783845d57d538eee7b99ababa6e2ebdcd620f40936ab"}
Sep 11 18:30:01.010455 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:01.010432    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp" event={"ID":"4e77659c-ef34-4072-bb36-4fee9101b297","Type":"ContainerStarted","Data":"bda7752f8a10faf2ed656f94ed28e838702bb908bc90c5441ac588d1c6f88c1d"}
Sep 11 18:30:01.011970 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:01.011943    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7" event={"ID":"711fa7dc-b20c-4cf1-be20-e7ff3ae94c45","Type":"ContainerStarted","Data":"88fefc2a1e78459e94cbaf316b28cfa56d2a12cf760a49019cf17aceba7d66dc"}
Sep 11 18:30:01.012060 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:01.011991    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7" event={"ID":"711fa7dc-b20c-4cf1-be20-e7ff3ae94c45","Type":"ContainerStarted","Data":"5c78bcb020818f15d3ce933c042b81d26bca75137c53a295ffa3556561b8f5dd"}
Sep 11 18:30:01.025664 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:01.025620    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp" podStartSLOduration=1.025606747 podStartE2EDuration="1.025606747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:01.024985173 +0000 UTC m=+2164.795051521" watchObservedRunningTime="2026-09-11 18:30:01.025606747 +0000 UTC m=+2164.795673097"
Sep 11 18:30:01.041916 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:01.041871    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7" podStartSLOduration=1.041857629 podStartE2EDuration="1.041857629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:01.039759785 +0000 UTC m=+2164.809826138" watchObservedRunningTime="2026-09-11 18:30:01.041857629 +0000 UTC m=+2164.811923976"
Sep 11 18:30:02.019375 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:02.019340    2571 generic.go:356] "Generic (PLEG): container finished" podID="4e77659c-ef34-4072-bb36-4fee9101b297" containerID="8941883c0c272db08f5a783845d57d538eee7b99ababa6e2ebdcd620f40936ab" exitCode=0
Sep 11 18:30:02.019764 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:02.019426    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp" event={"ID":"4e77659c-ef34-4072-bb36-4fee9101b297","Type":"ContainerDied","Data":"8941883c0c272db08f5a783845d57d538eee7b99ababa6e2ebdcd620f40936ab"}
Sep 11 18:30:02.020898 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:02.020879    2571 generic.go:356] "Generic (PLEG): container finished" podID="711fa7dc-b20c-4cf1-be20-e7ff3ae94c45" containerID="88fefc2a1e78459e94cbaf316b28cfa56d2a12cf760a49019cf17aceba7d66dc" exitCode=0
Sep 11 18:30:02.020979 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:02.020959    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7" event={"ID":"711fa7dc-b20c-4cf1-be20-e7ff3ae94c45","Type":"ContainerDied","Data":"88fefc2a1e78459e94cbaf316b28cfa56d2a12cf760a49019cf17aceba7d66dc"}
Sep 11 18:30:03.172524 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:03.172496    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7"
Sep 11 18:30:03.175202 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:03.175182    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"
Sep 11 18:30:04.034434 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.034408    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"
Sep 11 18:30:04.034621 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.034407    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp" event={"ID":"4e77659c-ef34-4072-bb36-4fee9101b297","Type":"ContainerDied","Data":"bda7752f8a10faf2ed656f94ed28e838702bb908bc90c5441ac588d1c6f88c1d"}
Sep 11 18:30:04.034621 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.034521    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bda7752f8a10faf2ed656f94ed28e838702bb908bc90c5441ac588d1c6f88c1d"
Sep 11 18:30:04.035822 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.035799    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7"
Sep 11 18:30:04.035943 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.035820    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819190-jjcv7" event={"ID":"711fa7dc-b20c-4cf1-be20-e7ff3ae94c45","Type":"ContainerDied","Data":"5c78bcb020818f15d3ce933c042b81d26bca75137c53a295ffa3556561b8f5dd"}
Sep 11 18:30:04.035943 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.035860    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c78bcb020818f15d3ce933c042b81d26bca75137c53a295ffa3556561b8f5dd"
Sep 11 18:30:04.264549 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.264519    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"]
Sep 11 18:30:04.269008 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.268983    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5d35e7-29819190-jphnp"]
Sep 11 18:30:04.290538 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.290472    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"]
Sep 11 18:30:04.290685 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.290662    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d" podUID="215a093e-eb0b-441d-b7b2-bcf9d911c846" containerName="payload-pre-processing" containerID="cri-o://db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e" gracePeriod=30
Sep 11 18:30:04.511277 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.511244    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"]
Sep 11 18:30:04.514519 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.514493    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5d35e7-kgm9n"]
Sep 11 18:30:04.543709 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.543659    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:30:04.650584 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.650545    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215a093e-eb0b-441d-b7b2-bcf9d911c846-plugins-config-volume\") pod \"215a093e-eb0b-441d-b7b2-bcf9d911c846\" (UID: \"215a093e-eb0b-441d-b7b2-bcf9d911c846\") "
Sep 11 18:30:04.650774 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.650652    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p8d4\" (UniqueName: \"kubernetes.io/projected/215a093e-eb0b-441d-b7b2-bcf9d911c846-kube-api-access-5p8d4\") pod \"215a093e-eb0b-441d-b7b2-bcf9d911c846\" (UID: \"215a093e-eb0b-441d-b7b2-bcf9d911c846\") "
Sep 11 18:30:04.650982 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.650956    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215a093e-eb0b-441d-b7b2-bcf9d911c846-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "215a093e-eb0b-441d-b7b2-bcf9d911c846" (UID: "215a093e-eb0b-441d-b7b2-bcf9d911c846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:04.652852 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.652824    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215a093e-eb0b-441d-b7b2-bcf9d911c846-kube-api-access-5p8d4" (OuterVolumeSpecName: "kube-api-access-5p8d4") pod "215a093e-eb0b-441d-b7b2-bcf9d911c846" (UID: "215a093e-eb0b-441d-b7b2-bcf9d911c846"). InnerVolumeSpecName "kube-api-access-5p8d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:04.730614 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.730583    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e77659c-ef34-4072-bb36-4fee9101b297" path="/var/lib/kubelet/pods/4e77659c-ef34-4072-bb36-4fee9101b297/volumes"
Sep 11 18:30:04.730978 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.730959    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f226e3e7-d39f-4ae4-9a56-294afd9c1f9b" path="/var/lib/kubelet/pods/f226e3e7-d39f-4ae4-9a56-294afd9c1f9b/volumes"
Sep 11 18:30:04.751349 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.751322    2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215a093e-eb0b-441d-b7b2-bcf9d911c846-plugins-config-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:30:04.751447 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:04.751352    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8d4\" (UniqueName: \"kubernetes.io/projected/215a093e-eb0b-441d-b7b2-bcf9d911c846-kube-api-access-5p8d4\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:30:05.043778 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.043746    2571 generic.go:356] "Generic (PLEG): container finished" podID="215a093e-eb0b-441d-b7b2-bcf9d911c846" containerID="db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e" exitCode=0
Sep 11 18:30:05.043951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.043797    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"
Sep 11 18:30:05.043951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.043836    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d" event={"ID":"215a093e-eb0b-441d-b7b2-bcf9d911c846","Type":"ContainerDied","Data":"db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e"}
Sep 11 18:30:05.043951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.043892    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d" event={"ID":"215a093e-eb0b-441d-b7b2-bcf9d911c846","Type":"ContainerDied","Data":"f41ea3dfd28b864e6cff8664a407419558f6527f2744db84d63d9ef423252bdb"}
Sep 11 18:30:05.043951 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.043922    2571 scope.go:117] "RemoveContainer" containerID="db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e"
Sep 11 18:30:05.054227 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.054186    2571 scope.go:117] "RemoveContainer" containerID="db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e"
Sep 11 18:30:05.054516 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:30:05.054499    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e\": container with ID starting with db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e not found: ID does not exist" containerID="db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e"
Sep 11 18:30:05.054587 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.054524    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e"} err="failed to get container status \"db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e\": rpc error: code = NotFound desc = could not find container \"db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e\": container with ID starting with db6795dde3f2ac1e0b635ed4dd6c4d4b061e5f95570415ad3003cfe8b0d59c4e not found: ID does not exist"
Sep 11 18:30:05.065585 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.065557    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"]
Sep 11 18:30:05.070544 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:05.070524    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5d35e7-84df4fb4b6-ddt8d"]
Sep 11 18:30:06.730475 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:30:06.730404    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215a093e-eb0b-441d-b7b2-bcf9d911c846" path="/var/lib/kubelet/pods/215a093e-eb0b-441d-b7b2-bcf9d911c846/volumes"
Sep 11 18:32:44.073704 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:44.073665    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"]
Sep 11 18:32:44.074179 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:44.073886    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc" podUID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerName="manager" containerID="cri-o://12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea" gracePeriod=10
Sep 11 18:32:45.524833 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.524810    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:32:45.627407 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.627371    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdw7z\" (UniqueName: \"kubernetes.io/projected/64419844-ea38-4dc6-b2bd-6b5e66eaa060-kube-api-access-gdw7z\") pod \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\" (UID: \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\") "
Sep 11 18:32:45.627407 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.627412    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64419844-ea38-4dc6-b2bd-6b5e66eaa060-extensions-socket-volume\") pod \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\" (UID: \"64419844-ea38-4dc6-b2bd-6b5e66eaa060\") "
Sep 11 18:32:45.627821 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.627797    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64419844-ea38-4dc6-b2bd-6b5e66eaa060-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "64419844-ea38-4dc6-b2bd-6b5e66eaa060" (UID: "64419844-ea38-4dc6-b2bd-6b5e66eaa060"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:32:45.629429 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.629409    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64419844-ea38-4dc6-b2bd-6b5e66eaa060-kube-api-access-gdw7z" (OuterVolumeSpecName: "kube-api-access-gdw7z") pod "64419844-ea38-4dc6-b2bd-6b5e66eaa060" (UID: "64419844-ea38-4dc6-b2bd-6b5e66eaa060"). InnerVolumeSpecName "kube-api-access-gdw7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:45.728348 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.728314    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdw7z\" (UniqueName: \"kubernetes.io/projected/64419844-ea38-4dc6-b2bd-6b5e66eaa060-kube-api-access-gdw7z\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:32:45.728348 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.728341    2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64419844-ea38-4dc6-b2bd-6b5e66eaa060-extensions-socket-volume\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:32:45.959222 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.959189    2571 generic.go:356] "Generic (PLEG): container finished" podID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerID="12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea" exitCode=0
Sep 11 18:32:45.959489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.959237    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"
Sep 11 18:32:45.959489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.959277    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc" event={"ID":"64419844-ea38-4dc6-b2bd-6b5e66eaa060","Type":"ContainerDied","Data":"12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea"}
Sep 11 18:32:45.959489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.959355    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc" event={"ID":"64419844-ea38-4dc6-b2bd-6b5e66eaa060","Type":"ContainerDied","Data":"d67627ebec798df384f2784dc12875f23dd9e789099195f746ecc2cb50061b0d"}
Sep 11 18:32:45.959489 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.959390    2571 scope.go:117] "RemoveContainer" containerID="12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea"
Sep 11 18:32:45.974119 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.974099    2571 scope.go:117] "RemoveContainer" containerID="448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98"
Sep 11 18:32:45.982486 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.982468    2571 scope.go:117] "RemoveContainer" containerID="12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea"
Sep 11 18:32:45.982801 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:32:45.982769    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea\": container with ID starting with 12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea not found: ID does not exist" containerID="12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea"
Sep 11 18:32:45.982856 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.982809    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea"} err="failed to get container status \"12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea\": rpc error: code = NotFound desc = could not find container \"12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea\": container with ID starting with 12857c658f74f26cb657eed8582fd1ee2254e14153adf3a0f251ec88063728ea not found: ID does not exist"
Sep 11 18:32:45.982856 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.982828    2571 scope.go:117] "RemoveContainer" containerID="448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98"
Sep 11 18:32:45.983085 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:32:45.983071    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98\": container with ID starting with 448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98 not found: ID does not exist" containerID="448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98"
Sep 11 18:32:45.983136 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.983087    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98"} err="failed to get container status \"448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98\": rpc error: code = NotFound desc = could not find container \"448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98\": container with ID starting with 448e76ae84e5f5277dd845ffb08f81f46e8896b064b3548c05fed5e8fc06fc98 not found: ID does not exist"
Sep 11 18:32:45.987212 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.987185    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"]
Sep 11 18:32:45.992752 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:45.992732    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rhbfc"]
Sep 11 18:32:46.729769 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:32:46.729738    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" path="/var/lib/kubelet/pods/64419844-ea38-4dc6-b2bd-6b5e66eaa060/volumes"
Sep 11 18:33:44.819585 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819549    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"]
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819873    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215a093e-eb0b-441d-b7b2-bcf9d911c846" containerName="payload-pre-processing"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819883    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="215a093e-eb0b-441d-b7b2-bcf9d911c846" containerName="payload-pre-processing"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819891    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerName="manager"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819897    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerName="manager"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819920    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e77659c-ef34-4072-bb36-4fee9101b297" containerName="cleanup"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819925    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e77659c-ef34-4072-bb36-4fee9101b297" containerName="cleanup"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819932    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711fa7dc-b20c-4cf1-be20-e7ff3ae94c45" containerName="cleanup"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819937    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="711fa7dc-b20c-4cf1-be20-e7ff3ae94c45" containerName="cleanup"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819985    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="711fa7dc-b20c-4cf1-be20-e7ff3ae94c45" containerName="cleanup"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819992    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerName="manager"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.819997    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="215a093e-eb0b-441d-b7b2-bcf9d911c846" containerName="payload-pre-processing"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.820003    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerName="manager"
Sep 11 18:33:44.820030 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.820010    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e77659c-ef34-4072-bb36-4fee9101b297" containerName="cleanup"
Sep 11 18:33:44.822930 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.822914    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:44.833235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.833214    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jpgrj\""
Sep 11 18:33:44.848189 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.848162    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"]
Sep 11 18:33:44.941190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.941154    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxm76\" (UniqueName: \"kubernetes.io/projected/9159b5db-0ad1-44f7-8e4c-0d715a0c970b-kube-api-access-lxm76\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kngcz\" (UID: \"9159b5db-0ad1-44f7-8e4c-0d715a0c970b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:44.941190 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:44.941194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9159b5db-0ad1-44f7-8e4c-0d715a0c970b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kngcz\" (UID: \"9159b5db-0ad1-44f7-8e4c-0d715a0c970b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:45.042275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:45.042233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxm76\" (UniqueName: \"kubernetes.io/projected/9159b5db-0ad1-44f7-8e4c-0d715a0c970b-kube-api-access-lxm76\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kngcz\" (UID: \"9159b5db-0ad1-44f7-8e4c-0d715a0c970b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:45.042275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:45.042279    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9159b5db-0ad1-44f7-8e4c-0d715a0c970b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kngcz\" (UID: \"9159b5db-0ad1-44f7-8e4c-0d715a0c970b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:45.042722 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:45.042699    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9159b5db-0ad1-44f7-8e4c-0d715a0c970b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kngcz\" (UID: \"9159b5db-0ad1-44f7-8e4c-0d715a0c970b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:45.056399 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:45.056370    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxm76\" (UniqueName: \"kubernetes.io/projected/9159b5db-0ad1-44f7-8e4c-0d715a0c970b-kube-api-access-lxm76\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kngcz\" (UID: \"9159b5db-0ad1-44f7-8e4c-0d715a0c970b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:45.132546 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:45.132458    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:45.411981 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:45.411954    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"]
Sep 11 18:33:45.414855 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:33:45.414824    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9159b5db_0ad1_44f7_8e4c_0d715a0c970b.slice/crio-f41d453976d7387bf7a69329e5ad1fb5981ffdfca0fd310771e3dcdccd58d38f WatchSource:0}: Error finding container f41d453976d7387bf7a69329e5ad1fb5981ffdfca0fd310771e3dcdccd58d38f: Status 404 returned error can't find the container with id f41d453976d7387bf7a69329e5ad1fb5981ffdfca0fd310771e3dcdccd58d38f
Sep 11 18:33:46.299916 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:46.299865    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz" event={"ID":"9159b5db-0ad1-44f7-8e4c-0d715a0c970b","Type":"ContainerStarted","Data":"05013ff2138e21ceddc7aed4b3eceee5d96b0a1ede0bf63497d78cdc5f259e75"}
Sep 11 18:33:46.300403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:46.299969    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:46.300403 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:46.299988    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz" event={"ID":"9159b5db-0ad1-44f7-8e4c-0d715a0c970b","Type":"ContainerStarted","Data":"f41d453976d7387bf7a69329e5ad1fb5981ffdfca0fd310771e3dcdccd58d38f"}
Sep 11 18:33:46.324929 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:46.324866    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz" podStartSLOduration=2.324850317 podStartE2EDuration="2.324850317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:46.324493231 +0000 UTC m=+2390.094559591" watchObservedRunningTime="2026-09-11 18:33:46.324850317 +0000 UTC m=+2390.094916667"
Sep 11 18:33:57.308275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:57.308242    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kngcz"
Sep 11 18:33:57.604969 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:33:57.604886    2571 scope.go:117] "RemoveContainer" containerID="dc9151f279dd6be068fbed09bbba5487e2024bfc5cebbfda22508cafec31d18a"
Sep 11 18:34:29.223605 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.223519    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"]
Sep 11 18:34:29.224174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.223867    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerName="manager"
Sep 11 18:34:29.224174 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.223879    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="64419844-ea38-4dc6-b2bd-6b5e66eaa060" containerName="manager"
Sep 11 18:34:29.227146 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.227127    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.230000 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.229985    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c891a30d-kserve-self-signed-certs\""
Sep 11 18:34:29.237570 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.237544    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"]
Sep 11 18:34:29.321797 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.321755    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-model-cache\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.321973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.321806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tmp-dir\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.321973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.321865    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tls-certs\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.321973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.321902    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmd9\" (UniqueName: \"kubernetes.io/projected/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-kube-api-access-mvmd9\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.321973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.321935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-dshm\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.321973 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.321965    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-home\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.422565 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.422526    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-home\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.422766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.422587    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-model-cache\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.422766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.422623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tmp-dir\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.422766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.422656    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tls-certs\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.422766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.422694    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmd9\" (UniqueName: \"kubernetes.io/projected/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-kube-api-access-mvmd9\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.422766 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.422732    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-dshm\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.423044 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.422909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-home\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.423044 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.422999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tmp-dir\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.423044 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.423026    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-model-cache\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.424940 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.424921    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-dshm\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.425208 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.425192    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tls-certs\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.431529 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.431503    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmd9\" (UniqueName: \"kubernetes.io/projected/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-kube-api-access-mvmd9\") pod \"e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") " pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.536020 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.535920    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:29.811651 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:29.811624    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"]
Sep 11 18:34:29.812654 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:34:29.812627    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ab3b77b_58c1_4869_88fe_bc910aa8c7a4.slice/crio-f84e231e6e64ff792010758284b909622c6491383c12b303088b19d5acf8572e WatchSource:0}: Error finding container f84e231e6e64ff792010758284b909622c6491383c12b303088b19d5acf8572e: Status 404 returned error can't find the container with id f84e231e6e64ff792010758284b909622c6491383c12b303088b19d5acf8572e
Sep 11 18:34:30.555362 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:30.555321    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd" event={"ID":"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4","Type":"ContainerStarted","Data":"02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306"}
Sep 11 18:34:30.555778 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:30.555375    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd" event={"ID":"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4","Type":"ContainerStarted","Data":"f84e231e6e64ff792010758284b909622c6491383c12b303088b19d5acf8572e"}
Sep 11 18:34:30.576740 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:30.576684    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd" podStartSLOduration=1.576667762 podStartE2EDuration="1.576667762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:30.575621928 +0000 UTC m=+2434.345688276" watchObservedRunningTime="2026-09-11 18:34:30.576667762 +0000 UTC m=+2434.346734136"
Sep 11 18:34:34.714387 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.714348    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"]
Sep 11 18:34:34.717598 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.717579    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.720562 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.720539    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c891a30d-kserve-self-signed-certs\""
Sep 11 18:34:34.732880 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.732858    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"]
Sep 11 18:34:34.769795 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.769751    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-home\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.769795 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.769793    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-dshm\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.769978 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.769813    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-model-cache\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.769978 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.769887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b1fe65-0db9-4b04-9761-293f39dc96a5-tls-certs\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.769978 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.769936    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnp9z\" (UniqueName: \"kubernetes.io/projected/56b1fe65-0db9-4b04-9761-293f39dc96a5-kube-api-access-xnp9z\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.769978 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.769973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-tmp-dir\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.870504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.870460    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b1fe65-0db9-4b04-9761-293f39dc96a5-tls-certs\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.870702 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.870520    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnp9z\" (UniqueName: \"kubernetes.io/projected/56b1fe65-0db9-4b04-9761-293f39dc96a5-kube-api-access-xnp9z\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.870702 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.870568    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-tmp-dir\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.870702 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.870626    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-home\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.870702 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.870649    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-dshm\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.870702 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.870676    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-model-cache\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.871197 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.871168    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-model-cache\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.871197 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.871188    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-tmp-dir\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.871411 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.871261    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-home\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.872957 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.872934    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-dshm\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.873161 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.873142    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b1fe65-0db9-4b04-9761-293f39dc96a5-tls-certs\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:34.881926 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:34.881900    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnp9z\" (UniqueName: \"kubernetes.io/projected/56b1fe65-0db9-4b04-9761-293f39dc96a5-kube-api-access-xnp9z\") pod \"e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") " pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:35.027586 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:35.027494    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:35.326015 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:35.325937    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"]
Sep 11 18:34:35.329130 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:34:35.329094    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56b1fe65_0db9_4b04_9761_293f39dc96a5.slice/crio-0999f3ba77e87b7d22a1f52dc4dc512b386715c779bd2d1533e27d90c2aa6103 WatchSource:0}: Error finding container 0999f3ba77e87b7d22a1f52dc4dc512b386715c779bd2d1533e27d90c2aa6103: Status 404 returned error can't find the container with id 0999f3ba77e87b7d22a1f52dc4dc512b386715c779bd2d1533e27d90c2aa6103
Sep 11 18:34:35.585950 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:35.585867    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r" event={"ID":"56b1fe65-0db9-4b04-9761-293f39dc96a5","Type":"ContainerStarted","Data":"763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489"}
Sep 11 18:34:35.585950 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:35.585908    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r" event={"ID":"56b1fe65-0db9-4b04-9761-293f39dc96a5","Type":"ContainerStarted","Data":"0999f3ba77e87b7d22a1f52dc4dc512b386715c779bd2d1533e27d90c2aa6103"}
Sep 11 18:34:35.627517 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:35.627468    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r" podStartSLOduration=1.627452879 podStartE2EDuration="1.627452879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:35.625944317 +0000 UTC m=+2439.396010675" watchObservedRunningTime="2026-09-11 18:34:35.627452879 +0000 UTC m=+2439.397519227"
Sep 11 18:34:39.536258 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:39.536220    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:39.536672 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:39.536276    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:39.550101 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:39.550073    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:39.622257 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:39.622229    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:34:45.028327 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:45.028262    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:45.028744 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:45.028348    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:45.040934 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:45.040908    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:45.654587 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:45.654558    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:34:49.106393 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:49.106359    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"]
Sep 11 18:34:50.037975 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:50.037940    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"]
Sep 11 18:34:57.626367 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:34:57.626335    2571 scope.go:117] "RemoveContainer" containerID="b23e7f31e2f3e0d3acf9220a7b183b5d43fc7ce23a7f0a985e9ea52f62969e01"
Sep 11 18:35:04.125211 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.125146    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd" podUID="9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" containerName="main" containerID="cri-o://02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306" gracePeriod=45
Sep 11 18:35:04.376186 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.376127    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:35:04.530214 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530178    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tls-certs\") pod \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") "
Sep 11 18:35:04.530214 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530222    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-home\") pod \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") "
Sep 11 18:35:04.530504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530242    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-dshm\") pod \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") "
Sep 11 18:35:04.530504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530263    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvmd9\" (UniqueName: \"kubernetes.io/projected/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-kube-api-access-mvmd9\") pod \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") "
Sep 11 18:35:04.530504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530385    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-model-cache\") pod \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") "
Sep 11 18:35:04.530504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530437    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tmp-dir\") pod \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\" (UID: \"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4\") "
Sep 11 18:35:04.530720 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530543    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-home" (OuterVolumeSpecName: "home") pod "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" (UID: "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:04.530720 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530621    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" (UID: "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:04.530796 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530745    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" (UID: "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:04.530876 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530858    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.530918 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530878    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.530918 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.530887    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.532548 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.532532    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-kube-api-access-mvmd9" (OuterVolumeSpecName: "kube-api-access-mvmd9") pod "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" (UID: "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4"). InnerVolumeSpecName "kube-api-access-mvmd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:04.532768 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.532743    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-dshm" (OuterVolumeSpecName: "dshm") pod "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" (UID: "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:04.532872 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.532743    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" (UID: "9ab3b77b-58c1-4869-88fe-bc910aa8c7a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:04.631588 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.631505    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.631588 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.631531    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.631588 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.631540    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvmd9\" (UniqueName: \"kubernetes.io/projected/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4-kube-api-access-mvmd9\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.749535 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.749495    2571 generic.go:356] "Generic (PLEG): container finished" podID="9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" containerID="02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306" exitCode=0
Sep 11 18:35:04.749715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.749536    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"
Sep 11 18:35:04.749715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.749571    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd" event={"ID":"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4","Type":"ContainerDied","Data":"02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306"}
Sep 11 18:35:04.749715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.749622    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd" event={"ID":"9ab3b77b-58c1-4869-88fe-bc910aa8c7a4","Type":"ContainerDied","Data":"f84e231e6e64ff792010758284b909622c6491383c12b303088b19d5acf8572e"}
Sep 11 18:35:04.749715 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.749655    2571 scope.go:117] "RemoveContainer" containerID="02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306"
Sep 11 18:35:04.762462 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.762435    2571 scope.go:117] "RemoveContainer" containerID="02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306"
Sep 11 18:35:04.762796 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:35:04.762772    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306\": container with ID starting with 02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306 not found: ID does not exist" containerID="02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306"
Sep 11 18:35:04.762877 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.762811    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306"} err="failed to get container status \"02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306\": rpc error: code = NotFound desc = could not find container \"02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306\": container with ID starting with 02c858ec753e38af8e1751031dedcbb945aced9eae19c9a4dd69ca963876d306 not found: ID does not exist"
Sep 11 18:35:04.770546 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.770519    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"]
Sep 11 18:35:04.779339 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:04.776514    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c891a30d-kserve-57b489dcd7-k87xd"]
Sep 11 18:35:05.060267 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.060205    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r" podUID="56b1fe65-0db9-4b04-9761-293f39dc96a5" containerName="main" containerID="cri-o://763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489" gracePeriod=45
Sep 11 18:35:05.307504 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.307480    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:35:05.438373 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438337    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-model-cache\") pod \"56b1fe65-0db9-4b04-9761-293f39dc96a5\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") "
Sep 11 18:35:05.438571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438390    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-home\") pod \"56b1fe65-0db9-4b04-9761-293f39dc96a5\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") "
Sep 11 18:35:05.438571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438430    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-dshm\") pod \"56b1fe65-0db9-4b04-9761-293f39dc96a5\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") "
Sep 11 18:35:05.438571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438473    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnp9z\" (UniqueName: \"kubernetes.io/projected/56b1fe65-0db9-4b04-9761-293f39dc96a5-kube-api-access-xnp9z\") pod \"56b1fe65-0db9-4b04-9761-293f39dc96a5\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") "
Sep 11 18:35:05.438571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438511    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-tmp-dir\") pod \"56b1fe65-0db9-4b04-9761-293f39dc96a5\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") "
Sep 11 18:35:05.438571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438534    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b1fe65-0db9-4b04-9761-293f39dc96a5-tls-certs\") pod \"56b1fe65-0db9-4b04-9761-293f39dc96a5\" (UID: \"56b1fe65-0db9-4b04-9761-293f39dc96a5\") "
Sep 11 18:35:05.438813 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438634    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-home" (OuterVolumeSpecName: "home") pod "56b1fe65-0db9-4b04-9761-293f39dc96a5" (UID: "56b1fe65-0db9-4b04-9761-293f39dc96a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:05.438813 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438654    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "56b1fe65-0db9-4b04-9761-293f39dc96a5" (UID: "56b1fe65-0db9-4b04-9761-293f39dc96a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:05.438813 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438766    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56b1fe65-0db9-4b04-9761-293f39dc96a5" (UID: "56b1fe65-0db9-4b04-9761-293f39dc96a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:05.438970 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438839    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-model-cache\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:05.438970 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.438858    2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-home\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:05.440543 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.440521    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b1fe65-0db9-4b04-9761-293f39dc96a5-kube-api-access-xnp9z" (OuterVolumeSpecName: "kube-api-access-xnp9z") pod "56b1fe65-0db9-4b04-9761-293f39dc96a5" (UID: "56b1fe65-0db9-4b04-9761-293f39dc96a5"). InnerVolumeSpecName "kube-api-access-xnp9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:05.440629 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.440615    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b1fe65-0db9-4b04-9761-293f39dc96a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56b1fe65-0db9-4b04-9761-293f39dc96a5" (UID: "56b1fe65-0db9-4b04-9761-293f39dc96a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:05.440675 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.440668    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-dshm" (OuterVolumeSpecName: "dshm") pod "56b1fe65-0db9-4b04-9761-293f39dc96a5" (UID: "56b1fe65-0db9-4b04-9761-293f39dc96a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:05.540069 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.540032    2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-dshm\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:05.540069 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.540063    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnp9z\" (UniqueName: \"kubernetes.io/projected/56b1fe65-0db9-4b04-9761-293f39dc96a5-kube-api-access-xnp9z\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:05.540284 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.540093    2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56b1fe65-0db9-4b04-9761-293f39dc96a5-tmp-dir\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:05.540284 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.540102    2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56b1fe65-0db9-4b04-9761-293f39dc96a5-tls-certs\") on node \"ip-10-0-133-109.ec2.internal\" DevicePath \"\""
Sep 11 18:35:05.756370 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.756272    2571 generic.go:356] "Generic (PLEG): container finished" podID="56b1fe65-0db9-4b04-9761-293f39dc96a5" containerID="763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489" exitCode=0
Sep 11 18:35:05.756370 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.756328    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"
Sep 11 18:35:05.756370 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.756345    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r" event={"ID":"56b1fe65-0db9-4b04-9761-293f39dc96a5","Type":"ContainerDied","Data":"763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489"}
Sep 11 18:35:05.756770 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.756387    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r" event={"ID":"56b1fe65-0db9-4b04-9761-293f39dc96a5","Type":"ContainerDied","Data":"0999f3ba77e87b7d22a1f52dc4dc512b386715c779bd2d1533e27d90c2aa6103"}
Sep 11 18:35:05.756770 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.756404    2571 scope.go:117] "RemoveContainer" containerID="763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489"
Sep 11 18:35:05.766136 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.766118    2571 scope.go:117] "RemoveContainer" containerID="763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489"
Sep 11 18:35:05.766401 ip-10-0-133-109 kubenswrapper[2571]: E0911 18:35:05.766383    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489\": container with ID starting with 763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489 not found: ID does not exist" containerID="763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489"
Sep 11 18:35:05.766462 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.766409    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489"} err="failed to get container status \"763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489\": rpc error: code = NotFound desc = could not find container \"763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489\": container with ID starting with 763df6687add3002a73cf84450a125e3cd4ed7531d1f0dfe0f9a4cfcd7f07489 not found: ID does not exist"
Sep 11 18:35:05.780514 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.780481    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"]
Sep 11 18:35:05.784628 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:05.784608    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c891a30d-kserve-64bbb4869f-fb49r"]
Sep 11 18:35:06.732690 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:06.732658    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56b1fe65-0db9-4b04-9761-293f39dc96a5" path="/var/lib/kubelet/pods/56b1fe65-0db9-4b04-9761-293f39dc96a5/volumes"
Sep 11 18:35:06.733072 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:06.733033    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" path="/var/lib/kubelet/pods/9ab3b77b-58c1-4869-88fe-bc910aa8c7a4/volumes"
Sep 11 18:35:36.837739 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:36.837706    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b7dcf54b-kvs4q_fd1aa1c1-771d-4055-b34a-b501595e51d4/authorino/0.log"
Sep 11 18:35:42.199387 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:42.199353    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nkr9j_edaf25f5-8c33-433d-b05c-1fb1b04bca25/manager/0.log"
Sep 11 18:35:42.312429 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:42.312394    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-66zsz_c959416b-dbbf-456e-83e6-b03b2a689e0b/manager/0.log"
Sep 11 18:35:42.551785 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:42.551710    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-thghb_a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43/server/0.log"
Sep 11 18:35:42.668236 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:42.668206    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zw2cp_c3c79da3-d8b5-4839-810d-66ce64aef094/manager/0.log"
Sep 11 18:35:42.916325 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:42.916283    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-fljr7_aba5fc8b-d00b-4dfd-be31-057d612f1539/copy-manifests/0.log"
Sep 11 18:35:42.922698 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:42.922681    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-fljr7_aba5fc8b-d00b-4dfd-be31-057d612f1539/manager/0.log"
Sep 11 18:35:43.608481 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:43.608453    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819175-j5lwl_4c044c58-ecd9-4ba2-a757-029e6975cfae/cleanup/0.log"
Sep 11 18:35:43.720409 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:43.720383    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819190-jjcv7_711fa7dc-b20c-4cf1-be20-e7ff3ae94c45/cleanup/0.log"
Sep 11 18:35:43.837943 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:43.837915    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-ktwm6_4b7a9066-b3da-4af9-8956-9c3d78c30f4d/postgres/0.log"
Sep 11 18:35:44.620456 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:44.620390    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b7dcf54b-kvs4q_fd1aa1c1-771d-4055-b34a-b501595e51d4/authorino/0.log"
Sep 11 18:35:44.745526 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:44.745495    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-f8bxx_3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e/manager/0.log"
Sep 11 18:35:44.856469 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:44.856440    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-k5xlt_0bd0932b-c6d5-4a02-837e-a96aca56a175/manager/0.log"
Sep 11 18:35:45.087912 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:45.087878    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-kngcz_9159b5db-0ad1-44f7-8e4c-0d715a0c970b/manager/0.log"
Sep 11 18:35:45.312739 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:45.312707    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dl7xx_2469f45e-5948-4024-8c59-a57a7ea18be2/manager/0.log"
Sep 11 18:35:45.989855 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:45.989810    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff57cc6b9-r4f9g_a3112d22-7697-4c4e-b898-0c8093d579cd/kube-auth-proxy/0.log"
Sep 11 18:35:46.108902 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:46.108852    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v4qdh_4dc54b69-92c8-4ed2-be6b-02651e084df7/istio-proxy/1.log"
Sep 11 18:35:46.348363 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:46.348274    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g2ncf_34cfa5cb-dbc3-4c99-903e-944f864cef32/payload-processing/0.log"
Sep 11 18:35:46.832555 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:46.832528    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-46txj_dd0f6b98-85bf-4087-aae4-6455f5af7568/main/0.log"
Sep 11 18:35:52.796899 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.796818    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mt88g/must-gather-d2mlv"]
Sep 11 18:35:52.797252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.797169    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56b1fe65-0db9-4b04-9761-293f39dc96a5" containerName="main"
Sep 11 18:35:52.797252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.797180    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="56b1fe65-0db9-4b04-9761-293f39dc96a5" containerName="main"
Sep 11 18:35:52.797252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.797191    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" containerName="main"
Sep 11 18:35:52.797252 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.797196    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" containerName="main"
Sep 11 18:35:52.797480 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.797260    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ab3b77b-58c1-4869-88fe-bc910aa8c7a4" containerName="main"
Sep 11 18:35:52.797480 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.797269    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="56b1fe65-0db9-4b04-9761-293f39dc96a5" containerName="main"
Sep 11 18:35:52.800863 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.800842    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt88g/must-gather-d2mlv"
Sep 11 18:35:52.803738 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.803714    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mt88g\"/\"openshift-service-ca.crt\""
Sep 11 18:35:52.803848 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.803744    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mt88g\"/\"default-dockercfg-2kcn4\""
Sep 11 18:35:52.804887 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.804871    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mt88g\"/\"kube-root-ca.crt\""
Sep 11 18:35:52.810843 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.810823    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt88g/must-gather-d2mlv"]
Sep 11 18:35:52.913109 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.913047    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrxw\" (UniqueName: \"kubernetes.io/projected/94bfc8fb-d7a0-4be7-9295-24edde972e15-kube-api-access-zjrxw\") pod \"must-gather-d2mlv\" (UID: \"94bfc8fb-d7a0-4be7-9295-24edde972e15\") " pod="openshift-must-gather-mt88g/must-gather-d2mlv"
Sep 11 18:35:52.913109 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:52.913124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94bfc8fb-d7a0-4be7-9295-24edde972e15-must-gather-output\") pod \"must-gather-d2mlv\" (UID: \"94bfc8fb-d7a0-4be7-9295-24edde972e15\") " pod="openshift-must-gather-mt88g/must-gather-d2mlv"
Sep 11 18:35:53.014236 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:53.014196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrxw\" (UniqueName: \"kubernetes.io/projected/94bfc8fb-d7a0-4be7-9295-24edde972e15-kube-api-access-zjrxw\") pod \"must-gather-d2mlv\" (UID: \"94bfc8fb-d7a0-4be7-9295-24edde972e15\") " pod="openshift-must-gather-mt88g/must-gather-d2mlv"
Sep 11 18:35:53.014236 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:53.014254    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94bfc8fb-d7a0-4be7-9295-24edde972e15-must-gather-output\") pod \"must-gather-d2mlv\" (UID: \"94bfc8fb-d7a0-4be7-9295-24edde972e15\") " pod="openshift-must-gather-mt88g/must-gather-d2mlv"
Sep 11 18:35:53.014605 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:53.014587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94bfc8fb-d7a0-4be7-9295-24edde972e15-must-gather-output\") pod \"must-gather-d2mlv\" (UID: \"94bfc8fb-d7a0-4be7-9295-24edde972e15\") " pod="openshift-must-gather-mt88g/must-gather-d2mlv"
Sep 11 18:35:53.024275 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:53.024244    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrxw\" (UniqueName: \"kubernetes.io/projected/94bfc8fb-d7a0-4be7-9295-24edde972e15-kube-api-access-zjrxw\") pod \"must-gather-d2mlv\" (UID: \"94bfc8fb-d7a0-4be7-9295-24edde972e15\") " pod="openshift-must-gather-mt88g/must-gather-d2mlv"
Sep 11 18:35:53.111150 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:53.111059    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt88g/must-gather-d2mlv"
Sep 11 18:35:53.434198 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:53.434175    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt88g/must-gather-d2mlv"]
Sep 11 18:35:53.435935 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:35:53.435907    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94bfc8fb_d7a0_4be7_9295_24edde972e15.slice/crio-04e97f8666c4f3576ae77882d623f48a545c2de2ab1c967490f072add0f68e15 WatchSource:0}: Error finding container 04e97f8666c4f3576ae77882d623f48a545c2de2ab1c967490f072add0f68e15: Status 404 returned error can't find the container with id 04e97f8666c4f3576ae77882d623f48a545c2de2ab1c967490f072add0f68e15
Sep 11 18:35:53.437871 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:53.437849    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:35:54.040599 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:54.040556    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/must-gather-d2mlv" event={"ID":"94bfc8fb-d7a0-4be7-9295-24edde972e15","Type":"ContainerStarted","Data":"04e97f8666c4f3576ae77882d623f48a545c2de2ab1c967490f072add0f68e15"}
Sep 11 18:35:55.048835 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:55.048804    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/must-gather-d2mlv" event={"ID":"94bfc8fb-d7a0-4be7-9295-24edde972e15","Type":"ContainerStarted","Data":"3efc9972a1155763d4d35999c4472d72822b9963f87e202c92f7572373de0079"}
Sep 11 18:35:55.048835 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:55.048847    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/must-gather-d2mlv" event={"ID":"94bfc8fb-d7a0-4be7-9295-24edde972e15","Type":"ContainerStarted","Data":"4b93ab545b11e5cb1c2488d17198cb78c5ddabc3369de897ec9bbc01f2ee8620"}
Sep 11 18:35:55.069830 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:55.069782    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mt88g/must-gather-d2mlv" podStartSLOduration=2.320181738 podStartE2EDuration="3.069766408s" totalImagesPullingTime="749.58467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:55.06767141 +0000 UTC m=+2518.837737757" watchObservedRunningTime="2026-09-11 18:35:55.069766408 +0000 UTC m=+2518.839832805"
Sep 11 18:35:55.960801 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:55.960772    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-snnhr_e2279fb0-d7d6-42b2-b779-c9ef928edc88/global-pull-secret-syncer/0.log"
Sep 11 18:35:56.099026 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:56.098992    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cnvwk_46bb7121-35c8-46ba-a6ff-885976cea1d1/konnectivity-agent/0.log"
Sep 11 18:35:56.180222 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:56.180192    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-109.ec2.internal_90fea5d757634326f5a2e77bf4cd7804/haproxy/0.log"
Sep 11 18:35:57.648332 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:57.648281    2571 scope.go:117] "RemoveContainer" containerID="7fb7ef01b861544cb6c01e1d4dbe09ac5334ded174418ff6e3839878588762bd"
Sep 11 18:35:57.664875 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:57.664850    2571 scope.go:117] "RemoveContainer" containerID="801b61cb25cb161691e933dca6b8c8841261041633084d1e41935ca64184c550"
Sep 11 18:35:57.680289 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:35:57.680255    2571 scope.go:117] "RemoveContainer" containerID="2c6df278560f7978eeb6e09197c022a35e5d2c6e334903710a99e5f23741681a"
Sep 11 18:36:00.667750 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:00.667633    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b7dcf54b-kvs4q_fd1aa1c1-771d-4055-b34a-b501595e51d4/authorino/0.log"
Sep 11 18:36:00.717982 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:00.717944    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-f8bxx_3ca65cf0-5e32-44d9-ac74-fb0e7f97c46e/manager/0.log"
Sep 11 18:36:00.747191 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:00.747131    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-k5xlt_0bd0932b-c6d5-4a02-837e-a96aca56a175/manager/0.log"
Sep 11 18:36:00.831195 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:00.831160    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-kngcz_9159b5db-0ad1-44f7-8e4c-0d715a0c970b/manager/0.log"
Sep 11 18:36:00.953474 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:00.953318    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dl7xx_2469f45e-5948-4024-8c59-a57a7ea18be2/manager/0.log"
Sep 11 18:36:03.124949 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:03.124856    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-snwm7_48aaac7f-0893-416d-99c3-f50d9aeb3ead/node-exporter/0.log"
Sep 11 18:36:03.150498 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:03.150467    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-snwm7_48aaac7f-0893-416d-99c3-f50d9aeb3ead/kube-rbac-proxy/0.log"
Sep 11 18:36:03.176129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:03.176095    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-snwm7_48aaac7f-0893-416d-99c3-f50d9aeb3ead/init-textfile/0.log"
Sep 11 18:36:05.682886 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.682851    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"]
Sep 11 18:36:05.688567 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.688545    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.703808 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.703774    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"]
Sep 11 18:36:05.766763 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.766723    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt89z\" (UniqueName: \"kubernetes.io/projected/914b761b-0347-4261-9c13-16c463422051-kube-api-access-zt89z\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.767088 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.767067    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-proc\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.767281 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.767262    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-lib-modules\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.767513 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.767495    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-sys\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.767661 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.767646    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-podres\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.868974    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-sys\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.869046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-podres\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.869090    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt89z\" (UniqueName: \"kubernetes.io/projected/914b761b-0347-4261-9c13-16c463422051-kube-api-access-zt89z\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.869143    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-proc\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.869196    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-lib-modules\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.869420    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-lib-modules\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.869482    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-sys\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.869556    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-podres\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.869817 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.869757    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/914b761b-0347-4261-9c13-16c463422051-proc\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:05.894934 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:05.894843    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt89z\" (UniqueName: \"kubernetes.io/projected/914b761b-0347-4261-9c13-16c463422051-kube-api-access-zt89z\") pod \"perf-node-gather-daemonset-5lnm8\" (UID: \"914b761b-0347-4261-9c13-16c463422051\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:06.003212 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:06.002886    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:06.298777 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:06.298735    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"]
Sep 11 18:36:06.302382 ip-10-0-133-109 kubenswrapper[2571]: W0911 18:36:06.302353    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod914b761b_0347_4261_9c13_16c463422051.slice/crio-c4209147b48ddf8b644a0f17f3a3733fe6fae9fddf8620557ac89bcc4585a0d1 WatchSource:0}: Error finding container c4209147b48ddf8b644a0f17f3a3733fe6fae9fddf8620557ac89bcc4585a0d1: Status 404 returned error can't find the container with id c4209147b48ddf8b644a0f17f3a3733fe6fae9fddf8620557ac89bcc4585a0d1
Sep 11 18:36:07.158629 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:07.158582    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8" event={"ID":"914b761b-0347-4261-9c13-16c463422051","Type":"ContainerStarted","Data":"8ac565eae026d8a393b83a493606f38ea4e1641d46e246b337e725b12432f1e4"}
Sep 11 18:36:07.159141 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:07.158657    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8" event={"ID":"914b761b-0347-4261-9c13-16c463422051","Type":"ContainerStarted","Data":"c4209147b48ddf8b644a0f17f3a3733fe6fae9fddf8620557ac89bcc4585a0d1"}
Sep 11 18:36:07.159469 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:07.159444    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:07.182318 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:07.182240    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8" podStartSLOduration=2.182219008 podStartE2EDuration="2.182219008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:07.179383888 +0000 UTC m=+2530.949450237" watchObservedRunningTime="2026-09-11 18:36:07.182219008 +0000 UTC m=+2530.952285357"
Sep 11 18:36:08.731850 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:08.731818    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lgnjx_e860867b-f1e8-4d20-a11b-bb01c144fdc2/dns/0.log"
Sep 11 18:36:08.755185 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:08.755155    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lgnjx_e860867b-f1e8-4d20-a11b-bb01c144fdc2/kube-rbac-proxy/0.log"
Sep 11 18:36:08.832342 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:08.832289    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l8kwh_c3685618-7216-470e-af33-3c64fb1618c4/dns-node-resolver/0.log"
Sep 11 18:36:09.501121 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:09.501077    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j6bg7_7b5f0490-e3a5-415d-a5b5-8702db1664b3/node-ca/0.log"
Sep 11 18:36:11.163038 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:11.162994    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff57cc6b9-r4f9g_a3112d22-7697-4c4e-b898-0c8093d579cd/kube-auth-proxy/0.log"
Sep 11 18:36:11.198945 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:11.198913    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v4qdh_4dc54b69-92c8-4ed2-be6b-02651e084df7/istio-proxy/0.log"
Sep 11 18:36:11.202312 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:11.202264    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v4qdh_4dc54b69-92c8-4ed2-be6b-02651e084df7/istio-proxy/1.log"
Sep 11 18:36:11.279448 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:11.279415    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g2ncf_34cfa5cb-dbc3-4c99-903e-944f864cef32/payload-processing/0.log"
Sep 11 18:36:12.279662 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:12.279634    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hn9nt_065a0aab-56f6-461e-bfb1-e9a51a830677/serve-healthcheck-canary/0.log"
Sep 11 18:36:13.165370 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:13.165339    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nnjnk_ed74f2d5-3e16-4910-8e84-12da8638b9a6/kube-rbac-proxy/0.log"
Sep 11 18:36:13.189744 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:13.189719    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nnjnk_ed74f2d5-3e16-4910-8e84-12da8638b9a6/exporter/0.log"
Sep 11 18:36:13.215794 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:13.215758    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nnjnk_ed74f2d5-3e16-4910-8e84-12da8638b9a6/extractor/0.log"
Sep 11 18:36:14.183646 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:14.183618    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-5lnm8"
Sep 11 18:36:15.536011 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:15.535984    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-nkr9j_edaf25f5-8c33-433d-b05c-1fb1b04bca25/manager/0.log"
Sep 11 18:36:15.909891 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:15.909860    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-66zsz_c959416b-dbbf-456e-83e6-b03b2a689e0b/manager/0.log"
Sep 11 18:36:15.989767 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:15.989690    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-thghb_a5a91f0f-8a3f-44b8-b8fb-7820efbb5a43/server/0.log"
Sep 11 18:36:16.100622 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:16.100588    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-zw2cp_c3c79da3-d8b5-4839-810d-66ce64aef094/manager/0.log"
Sep 11 18:36:16.283036 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:16.283011    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-fljr7_aba5fc8b-d00b-4dfd-be31-057d612f1539/manager/0.log"
Sep 11 18:36:16.311021 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:16.310993    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-fljr7_aba5fc8b-d00b-4dfd-be31-057d612f1539/copy-manifests/0.log"
Sep 11 18:36:16.797163 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:16.797127    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff57cc6b9-r4f9g_a3112d22-7697-4c4e-b898-0c8093d579cd/kube-auth-proxy/0.log"
Sep 11 18:36:16.827235 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:16.827174    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v4qdh_4dc54b69-92c8-4ed2-be6b-02651e084df7/istio-proxy/0.log"
Sep 11 18:36:16.830589 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:16.830562    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v4qdh_4dc54b69-92c8-4ed2-be6b-02651e084df7/istio-proxy/1.log"
Sep 11 18:36:16.889624 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:16.889585    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g2ncf_34cfa5cb-dbc3-4c99-903e-944f864cef32/payload-processing/0.log"
Sep 11 18:36:17.996345 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:17.996311    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-79h58_4a21436e-4ff5-4b36-b48d-fb77f3b84f04/openshift-lws-operator/0.log"
Sep 11 18:36:19.145078 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:19.145042    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff57cc6b9-r4f9g_a3112d22-7697-4c4e-b898-0c8093d579cd/kube-auth-proxy/0.log"
Sep 11 18:36:19.168435 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:19.168406    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v4qdh_4dc54b69-92c8-4ed2-be6b-02651e084df7/istio-proxy/0.log"
Sep 11 18:36:19.171571 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:19.171554    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-v4qdh_4dc54b69-92c8-4ed2-be6b-02651e084df7/istio-proxy/1.log"
Sep 11 18:36:19.247288 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:19.247211    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-g2ncf_34cfa5cb-dbc3-4c99-903e-944f864cef32/payload-processing/0.log"
Sep 11 18:36:23.269566 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:23.269539    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-m8gqp_dd90b17f-50cf-480d-951d-42bf44b98377/migrator/0.log"
Sep 11 18:36:23.296459 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:23.296430    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-m8gqp_dd90b17f-50cf-480d-951d-42bf44b98377/graceful-termination/0.log"
Sep 11 18:36:23.704386 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:23.704328    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-bzsls_14a35285-829b-490a-a879-8b415812e26a/kube-storage-version-migrator-operator/1.log"
Sep 11 18:36:23.705129 ip-10-0-133-109 kubenswrapper[2571]: I0911 18:36:23.705111    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-bzsls_14a35285-829b-490a-a879-8b415812e26a/kube-storage-version-migrator-operator/0.log"